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SUMMARY 


'^«»ry^K. '«.«?'  cry  -t*  ~V:^ 


As  an  aspect  of  the  operational  decision  aid  program  of  the  Office 
of  Naval  Research,  the  second  in  a  series  of  decision  aiding  tools,  an 
/emission  control,  electronic  warfare  (EWAR)  decision  aid  was  evaluated. 
The  evaluation  objectively  tested  both  the  general  and  the  specific  utility 
of  the  aid. 

Description  of  the  EWAR  Aid 

The  EWAR  decision  aid  was  developed  by  Decision  Science  Ap¬ 
plications,  Inc. ,  and  is  representative  of  a  class  of  decision  aids  which 
stress  assessment  of  various  contingencies  and  trade  offs  among  con¬ 
flicting  alternatives.  The  version  tested  was  not  considered  to  be  ready 
for  use  but  had  reached  a  sufficient  developmental  stage  as  to  warrant 
some  evaluation  for  effectiveness  and  utility. 

Essentially,  the  EWAR  aid  evaluates  candidate  emission  control 
plans  in  a  hierarchical  manner  along  a  number  of  dimensions.  Four  class¬ 
es  of  information  are  made  available  by  the  aid  to  its  user:  (1)  radar 
order-of-battle,  (2)  surveillance  effectiveness,  (3)  information  denial, 
and  (4)  trade  off  analyses. 

The  first  information  class,  the  order-of-battle  information,  pro¬ 
vides  data  of  a  basic  nature  such  as  the  radar  suite  on  each  ship  in  the 
task  force  and  the  configuration  of  the  task  force.  The  second  and  third 
information  classes  provide  data  which  make  increasingly  abstract  as¬ 
sessments  of  potential  emission  plans  from  unique  and  somewhat  diver¬ 
gent  perspectives;  the  perspective  of  the  former  is  that  of  surveillance 
effectiveness  while  that  of  the  latter  is  information  denial.  The  final  class 
of  information  attempts  a  synthesis  or  a  trade  off  between  these  divergent 
perspectives.  It  evaluates  by  making  a  variety  of  assessments  of  the  in¬ 
terplay  of  the  surveillance  coverage  and  information  denial  aspects  of 
candidate  emission  control  plans. 

One  user  oriented  feature  of  the  aid  appears  to  be  especially  use¬ 
ful  to  the  fleet  user.  This  feature  allows  the  user  to  assign  a  "value"  to 
task  force  ships  which  represents  his  evaluation  of  the  various  ships' 
importance  to  a  specific  mission. 

Method 


Two  study  groups  were  established.  One  group  received  specific 
training  in  emission  control  planning  and  in  the  use  of  the  aid.  The  second 
group  received  only  training  in  the  use  of  the  aid.  Participants  in  both 
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groups  were  military  members  of  the  faculty  of  the  U.  S.  Naval  Academy. 
Both  groups  developed  EMCON  plans  with  and  without  the  use  of  the  EWAR 
aid.  Both  "easy"  and  "hard"  problems  were  involved.  Four  hypotheses 
were  tested;  (1)  emission  control  plans  produced  using  the  decision  aid 
will  be  superior  to  those  plans  produced  without  the  aid,  (2)  emission 
control  training  will  further  enhance  the  quality  of  the  plans,  (3)  the  dif¬ 
ficulty  of  the  problems  will  not  affect  the  quality  of  the  emission  control 
plans  generated  in  the  aided  condition,  and  (4)  there  will  be  a  high  cor¬ 
relation  between  constructed  problem  difficulty  and  perceived  problem 
difficulty. 

Data  pertinent  to  each  of  the  four  hypotheses  were  collected  and 
compared  on  the  basis  of  an  internal  criterion  measure --fraction  of  task 
force  remaining  after  a  strike.  The  criterion  measure  was  selected  be¬ 
cause  it  reflected  the  essential  and  divergent  aspects  of  emission  control 
planning  and  because  it  was  operationally  meaningful. 

Each  participant  was  also  interviewed  to  obtain  his  perceptions  of 
the  various  features  of  the  aid  and  his  opinions  about  how  completely  the  aid 
achieved  its  goals. 

Findings 

The  results  support  contentions  favoring  the  value  of  the  aid.  There 
was  a  positive  and  statistically  significant  effect  of  aiding  on  emission  con¬ 
trol  plan  quality.  The  merit  of  the  aid  can  be  indicated  in  terms  of  the 
value  of  the  task  force  remaining  after  a  simulated  air  strike.  The  use  of 
the  aid  led  to  approximately  a  25  percent  increase  in  this  value. 

A  multiple  regression  analyses  indicated  that  the  strength  of  the  aid 
was  largely  due  to  the  various  trade  off  evaluations  that  the  aid  provides. 

A  structural  equation  analysis  suggested  that  the  trade  off  was;  (1)  achieved 
by  a  reciprocal  activity  between  the  surveillance  effectiveness  and  informa¬ 
tion  denial,  and  (2)  most  heavily  a  function  of  surveillance  coverage. 

A  multiple  attribute  utility  analysis  suggested  that,  on  the  overall, 
the  aid  achieved  its  goals  quite  well. 

Interview  information  produced  evidence  that  the  participants  in 
the  study  believed  the  "second  order"  (e.g.,  trade  off)  information  provided 
by  the  aid  to  be  most  useful  while  the  more  basic  situational  information 
was  said  to  be  least  useful. 

No  statistically  significant  effects  were  noted  of  training  or  of 
problem  difficulty  on  EMCON  plan  quality. 
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I.  INTRODUCTION 


The  Office  of  Naval  Research  has  been  involved  for  the  past  five 
years  in  a  program  aimed  at  developing  technologies  for  operational  de¬ 
cision  aids  to  be  used  by  task  force  level  officers  and  their  staffs.  With¬ 
in  this  program,  the  Applied  Psychological  Services  has  been  concerned 
with  the  empirical  evaluation  of  various  aids  as  they  reach  a  suitable  de¬ 
velopmental  stage.  This  report  presents  the  results  of  the  second  such 
evaluation  undertaken  by  Applied  Psychological  Services. 

The  report  is  concerned  with  the  Electronic  Warfare  (EWAR)  de¬ 
cision  aid  under  development  by  Decision  Science  Applications,  Inc.  (Noble, 
et  al.  ,  1978).  The  EWAR  aid  was  designed  to  be  used  by  electronic  warfare 
officers  to  facilitate  the  development  and  appraisal  of  emission  control 
(EMCON)  plans. 

The  task  of  an  electronic  warfare  officer,  in  terms  of  radar  emissions, 
is  twofold.  The  two  aspects  of  EMCON  planning  which  he  must  consider  are, 
to  some  extent,  diametrically  opposed.  The  basic  function  of  radar  emitters 
is  to  provide  surveillance  coverage  against  possible  enemy  attacks,  either 
surface  or  air.  However,  active  radar  emitters  can  give  an  enemy  valuable 
information  which  can  be  used  for  locating,  identifying,  and  targeting  the 
ships  of  one's  own  task  force.  Therefore,  two  dimensions  of  EMCON  plan¬ 
ning  must  be  simultaneously  considered  and  traded  off:  (1)  the  need  for 
surveillance  coverage,  and  (2)  information  denial.  Of  course,  the  adequate 
balancing  or  weighting  will  vary  with  the  task  force,  its  mission,  the  con¬ 
figuration  of  the  task  force,  and  the  number  and  type  of  possible  enemy 
threats. 

EWAR  Decision  Aid 


The  EWAR  decision  aid  does  not  create  EMCON  plans  but  rather  the  aid 
provides  assessments  of  EMCON  plans  developed  by  the  user.  The  information 
yielded  by  the  aid  can  be  roughly  classified  into  four  groups;  (1)  order-of-battle, 
(2)  surveillance  effectiveness,  (3)  information  denial,  and  (4)  trade-off  analyses. 
The  EMCON  plan  a  user  wishes  to  assess  is  entered  into  the  computer  based  aid 
by  means  of  an  interactive  computer  terminal.  Once  the  EMCON  plan  is  entered, 
it  can  be  assessed  by  use  of  a  variety  of  commands  which  call  displays--either 
tables  or  graphs- -containing  information  pertinent  to  some  specific  aspect  of  the 
plan.  The  displays  are  made  available  on  cathode  ray  tubes  of  the  terminals  and 
a  "hard  copy"  capability  is  also  provided.  In  the  present  evaluation  of  the  EWAR 
aid,  only  a  subset  of  the  possible  displays  was  made  available.  Displays  consid¬ 
ered  by  the  aid's  developers  to  be  not  particularly  useful  within  the  context  of  a 
laboratory  evaluation  were  not  made  available.  Each  display  included  in  the 
evaluation  of  the  EWAR  aid  is  described  below. 
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Order -of- Battle.  Three  order -of-battle  displays 
are  presented;  (1)  list  of  the  task  force  ships  and  the 
value  of  each  ship,  (2)  the  position  of  each  ship  in  the 
convoy,  and  (3)  the  type  and  status  of  each  available 
radar  on  each  ship.  The  ship/  value  presentation  is 
illustrated  in  Exhibit  1.  Under  normal  use  of  the  aid, 
the  user  assigns  the  ship  value  as  a  function  of  the  rel¬ 
ative  importance  of  each  ship  to  the  mission.  However, 
for  purposes  of  this  study,  the  value  was  preassigned. 

The  value  assigned  to  a  ship  represents  its  normalized 
displacement  in  relation  to  a  baseline  ship  which  is  as¬ 
signed  a  value  of  1.  Also  displayed  are  the  ship  index 
numbers,  uniquely  assigned  by  the  aid  to  each  ship  in 
the  data  base. 

The  second  order -of-battle  display,  showing  the 
disposition  of  the  task  force,  is  illustrated  in  Exhibit  2. 
Each  ship  in  the  task  force  is  redundantly  represented 
by  a  pictorial  symbol  and  an  index  number.  The  size 
of  the  pictorial  representation  of  a  ship  type  to  some 
extent  correlates  with  its  value.  At  the  top  right  cor¬ 
ner  of  the  display,  two  sets  of  tables  relate  the  symbols 
to  ship  type,  and  the  task  force  ships  and  index  num¬ 
bers  to  the  appropriate  symbols. 

The  final  order -of-battle  display  (Exhibit  3)  pre¬ 
sents  the  EMCON  plan  under  consideration  (current 
plan).  This  is  essentially  a  matrix  of  task  force  ships 
and  radars.  An  "X"  in  a  row -column  interset  indi¬ 
cates  that  the  radar  type  (column)  is  aboard  the  indi¬ 
cated  ship  (row).  If  the  "X"  is  solid,  the  radar  is  emit¬ 
ting  and,  if  the  "X"  is  broken,  the  radar  is  not  emitting. 
Under  each  radar,  the  type  is  indicated;  S  =  Surface, 

A  =  Air. 

Surveillance  Displays.  The  previously  discussed 
order -of-battle  displays  organize  and  present  basic  in¬ 
formation  without  transforming  the  data  in  any  way.  In 
most  other  displays,  an  algorithm  is  applied  which  ab¬ 
stracts  and  assesses  various  features  of  an  EMCON 
plan.  For  example,  the  surveillance  displays  provide 
an  assessment  of  a  plan  in  terms  of  surveillance  effec¬ 
tiveness. 

Four  displays  supply  information  on  various  aspects 
of  the  surveillance  coverage.  The  first,  the  surveillance 
range  display,  contains  information  on  the  maximum  de¬ 
tection  range  for  each  radar  type  against  several  potential 
threats  (Exhibit  4).  Maximum  range  is  estimated  by  an 
algorithm  that  considers  the  altitude  of  the  threat,  its  ra¬ 
dar  cross  section,  and  its  velocity.  Maximum  detection 
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INDEX 

SHIP 

VALUE 

1 

KITTY  HAWK 

940. 000 

2 

SPRUANCE 

91. 000 

3 

NASTY 

1.000 

4 

TRUETT 

48. 000 

5 

CHICAGO 

203.000 

6 

OKLAHOMA  CITY 

175.000 

10 

HOLT  HE 

48.  000 

Exhibit  1.  The  list  of  task  force  ships,  their  index  numbers, 
and  their  values. 
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DISPOSITION  OF  TASK  FORCE 
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Exhibit  2.  The  disposition  of  the  task  force  centered  on  the  Kitty  Hawk.* 

♦This  .Exhibit,  as  well  as  Exhibits  3  through  20,  is  a  reduced,  but  exact, 
reproduction  of  the  actual  display. 
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^Because  all  radars  are  active  in  this 
example,  no  broken  Xs  are  shown. 
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MAXIMUM  DETECTION  RANGE  IN  NMI 
STREAKER  SNEAKER 
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Maximum  detection  ranges  for  each  radar  type 
against  two  hypothetical  threats.  Streaker  and 
Sneaker. 


6 


a 


f-  i 


range  for  each  radar  type  against  a  variety  of  threats 
is  contained  within  the  data  base.  However,  for  the 
purposes  of  the  evaluation,  two  hypothetical  threats 
were  used;  (1)  Sneaker,  which  possesses  the  charac¬ 
teristics  of  a  low,  slow  flying  missile,  and  (2)  Streak¬ 
er,  which  has  the  characteristics  of  a  high,  fast  flying 
missile.  The  maximum  detection  ranges  in  Exhibit  4 
were  obtained  by  averaging  across  the  ship  types  (in 
the  data  base)  which  contain  a  specific  radar  type. 

This  was  necessary  because  the  actual  range  varies 
with  the  height  of  the  radar  platform  on  a  ship's 
superstructure. 

The  second  surveillance  range  display,  illustrated 
in  Exhibit  5  for  Sneaker  and  in  Exhibit  6  for  Streaker, 
presents  the  maximum  detection  range  in  pictorial  form. 

Each  ship  in  the  task  force  is  indicated  by  a  symbol  ap¬ 
propriate  to  its  type  and  a  unique  index  number.  In  both 
exhibits,  no  circles  for  the  surface  radars  (SPS-10, 

SPS-55)  are  included  because  the  potential  threats  are 
both  aerial. 

The  third  surveillance  display  makes  use  of  the 
detection  ranges  to  calculate  detection  probabilities  and 
plots  the  probabilities  cumulatively.  The  cumulative 
detection  probabilities,  as  illustrated  in  Exhibit  7  for 
Sneaker  and  in  Exhibit  8  for  Streaker,  are  plotted  for 
three  detection  contours  with  probabilities  of;  0.  30, 

0. 50,  and  0.  90. 

The  information  contained  in  the  final  display  goes 
beyond  the  detection  probability  and  presents  interception 
probability  for  the  relevant  threats.  The  interception 
probability  (Exhibit  9)  is  given  for  the  current  EMCON 
plan  and  for  a  plan  in  which  all  the  radars  are  active  (ALL 
ON). 

The  mean  interception  probability  is  a  single  number 
reference--the  surveillance  score--which  indicates  the 
merit  of  an  EMCON  plan  in  terms  of  surveillance  coverage 
across  threat  types.  The  highest  mean  interception  proba¬ 
bility  or  highest  surveillance  score  that  can  be  obtained  for 
a  given  scenario  is  the  plan  in  which  all  radars  are  emitting. 
This  highest  probability  is  indicated  by  the  ALL  ON  plan  in 
Exhibit  9.  How  much  Loss  of  surveillance  coverage  results 
from  turning  some  radars  off  may  be  estimated  by  comparing 
the  interception  probabilities  of  the  current  plan  with  that  of 
the  ALL  ON  plan.  The  interception  probability  (Exhibit  9) 
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Exhibit  5.  Maximum  range  circles  for  air  search  radars  against  Sneaker  threat. 
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Exhibit  7.  The  cumulative  probability  of  detection  for  the  air  search 
radars  against  Sneaker  threat. 
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Exhibit  9.  The  surveillance  scores--the  interception 
probabilities  for  both  the  current  plan  and 
the  ALL  ON  plan. 


12 


t 


(for  an  EMCON  plan)  against  one  threat  type  was  a 
weighted  mean  of  the  individual  interception  proba¬ 
bilities  for  all  task  force  ships.  The  weights  were 
the  ships'  values. 


Information  Denial.  One  of  the  problems  facing 
an  EW  officer  in  developing  and  assessing  EMCON  plans 
is  ascertaining  how  much  information  is  being  given 
away.  EWAR  assumes  that  an  enemy  will  possess  some 
information  regardless  of  whether  or  not  any  emissions 
are  taking  place.  The  aid  assumes  that  the  enemy  knows 
the  identity  of  the  ships  and  the  location  of  the  task  force. 
It  also  assumes  that  the  enemy  is  actively  monitoring  the 
task  force  and  that  it  has  the  equipment  and  capability 
of:  (1)  identifying  the  emissions  of  any  particular  type 
of  radar,  and  (2)  identifying  the  location  of  the  emission. 
The  information  the  enemy  lacks,  and  that  which  an  EW 
officer  would  want  to  deny,  is  the  true  identity  of  each 
target.  That  is,  the  true  ship  assignments  are  the  only 
important  information  the  enemy  lacks  and  this  informa¬ 
tion  may  be  deduced  from  radar  emissions  with  varying 
degrees  of  success. 

The  EWAR  aid  further  assumes  that  the  enemy  will 
allocate  weapons  in  proportion  to  each  ship's  perceived 
value  and  that,  if  the  enemy  can  correctly  identify  each 
ship,  it  will  target  in  proportion  to  the  true  value  of  the 
ship. 


In  denying  information,  the  major  consideration 
is  how  to  reduce  the  probability  that  a  target  will  be 
correctly  identified.  The  effects  of  deactivating  radars 
or  sets  of  radars  that  uniquely  identify  a  ship  are  to  de¬ 
crease  the  probability  of  correctly  identifying  the  ship. 
This,  in  turn,  could  affect  the  perceived  value  of  the 
task  force's  ships  and  therefore  the  allocation  of  threats 
to  ships. 

The  EWAR  decision  aid  attempts  to  help  with  the 
information  denial  aspects  of  EMCON  planning  by  sup¬ 
plying  displays  that  provide  increasingly  complex  meas¬ 
ures  or  assessments  of  information  denial.  Three  of 
these  displays  were  used  in  the  EWAR  evaluation.  At 
the  lowest  level,  the  aid  supplies  details  of  the  prob¬ 
ability  that  a  radar  target  will  be  identified  as  a  partic¬ 
ular  ship.  Exhibit  10  illustrates  the  assignment  of 
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Exhibit  10.  The  ships  that  a  blip  could  be  identified  as  and  the 
probability  of  identification  as  that  ship. 
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radar  targets  to  ships.  The  first  column  contains  the 
name  of  the  actual  ship  producing  the  signature;  the  sec¬ 
ond  column  presents  the  ships  for  which  it  could  be  mis¬ 
taken,  and  the  third  column  presents  the  probability  of 
the  correct  and  the  mistaken  assignments.  Accordingly, 
the  Kitty  Hawk  could  be  identified  as  the  Kitty  Hawk  or 
Oklahoma  City  with  a  probability  of  0.50.  The  reason  for 
this  is  that  the  target  corresponding  to  the  Kitty  Hawk  on 
the  enemy's  radar  screen  has  four  radar  types  associated 
with  it  (see  Exhibit  3);  SPS-10,  SPS-30,  SPS-43,  and 
SPS-52.  Because  the  Oklahoma  City  also  has  these  same 
emitters,  the  Kitty  Hawk  could  be  misidentified  by  the 
enemy  as  the  Oklahoma  City. 

On  the  other  hand,  the  Chicago  is  uniquely  identified 
because  it  possesses  the  only  SPS-48  radar  in  the  task 
force. 


The  second  information  denial  display  (Exhibit  11) 
combines  the  ship  assignment  probability  with  the  value 
of  the  ship  to  produce  an  approximation  of  the  perceived 
targeting  value  of  the  ship.  The  first  column  of  Exhibit 
11  presents  the  ships  in  the  task  force.  The  second  col¬ 
umn  shows  the  value  of  each  ship.  The  bars  on  the  right 
indicate  the  true  and  perceived  fractional  values.  The 
true  value  of  the  Kitty  Hawk  is  940  and  constitutes  about 
62  percent  of  the  total  value  of  this  task  force.  However, 
because  the  Kitty  Hawk  and  the  Oklahoma  City  cannot  be 
distinguished  from  each  other,  the  perceived  value  of  the 
former  is  reduced  and  that  of  latter  increased  from  the 
true  values.  This  means  that  if  tai’geting  is  proportional 
to  perceived  value,  approximately  74  percent  of  the  enemy 
weapons  will  be  distributed  equally  between  the  Kitty  Hawk 
and  the  Oklahoma  City. 

The  final  information  denial  display  contains  a  set 
of  single  measures,  one  of  which  is  intended  as  an  infor¬ 
mation  denial  score.  This  display,  (Exhibit  12),  contains 
three  scores;  (1)  a  score  for  an  EMCON  plan  with  all  emit¬ 
ters  off,  (2)  a  score  for  the  current  EMCON  plan  (the  in¬ 
formation  denial  score),  and  (3)  a  score  for  an  EMCON 
plan  with  all  emitters  on.  Each  score  represents  the  frac¬ 
tional  reduction  in  the  amount  of  task  force  damage  when 
compared  to  the  task  force  damage  resulting  from  strikes 
by  Sneaker  and  Streaker  distributed  in  proportion  to  the 
true  value  of  the  ships. 
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Exhibit  11.  The  perceived  and  the  true  value  of  the  task  force's  ships. 
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Exhibit  12.  Information  denial  score  from  current  plan  and  for  plans 
with  "all  emitters  off"  and  "all  emitters  on.  " 
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In  the  Exhibit  12  example,  an  allocation  of  threats 
on  the  basis  of  the  ALL  ON  plan  and  the  current  plan 
would  represent  a  16  percent  reduction  in  expected  task 
force  damage.  This  would  be  due  to  the  inability  of  the 
enemy  to  identify  accurately  the  Kitty  Hawk  (Exhibit  10). 
Task  force  damage  would  be  further  reduced  by  elimi¬ 
nating  any  targeting  information  provided  by  the  emitters. 
This  could  be  accomplished  by  turning  all  emitters  off. 

In  the  Exhibit  12  example,  having  all  emitters  off  result¬ 
ed  in  an  information  denial  score  of  42. 

However,  it  is  possible  to  produce  an  EMCON  plan 
which  would  have  a  higher  information  denial  score  than 
the  ALL  OFF  plan.  This  could  be  accomplished  by  ma¬ 
nipulating  the  emitters  so  that  the  high  value  ship  is  mis- 
identified  as  a  low  value  ship  and  the  intermediate  value 
ships  are  misidentified  as  the  high  value  ship.  In  this 
case,  the  expected  damage  to  the  task  force  would  be 
lower  than  with  an  ALL  OFF  plan  because  the  majority 
of  the  threats  would  be  directed  toward  the  intermediate 
value  ships  while  few  or  none  would  be  allocated  to  the 
high  value  ship. 

Trade  Off  Analysis.  Whereas  the  surveillance  ef¬ 
fectiveness  and  information  denial  displays  may  be  em¬ 
ployed  to  evaluate  EMCON  plans  from  unique  and  diver¬ 
gent  perspectives,  the  trade  off  analysis  attempts  to  pre¬ 
sent  a  synthesis  of  these  divergent  views.  The  first  of 
the  trade  off  displays  combines  data  from  two  sources 
which  were  previously  described.  One  source  is  the  cum¬ 
ulative  detection  probability  (a  measure  of  surveillance 
effectiveness)  and  the  second  is  the  perceived  value  of 
the  targets  (a  measure  of  information  denial).  These  are 
combined  into  a  trade  off  map  as  illustrated  in  Exhibit  13. 
The  targeting  value  data  are  coded  in  two  forms.  One 
form  relates  to  each  ship's  true  value  while  the  second 
relates  to  each  ship's  perceived  value.  At  the  location 
of  each  ship,  the  upper  symbol  (a  heavy  line  or  ship) 
represents  the  true  value  of  the  ship  whereas  the  lower, 
light  line  represents  the  perceived  value  of  the  ship. 

Since  it  is  logical  that  threats  will  be  targeted  in  propor¬ 
tion  to  perceived  value,  the  ships  indexed  by  the  1  (the 
Kitty  Hawk)  and  the  6  (Oklahoma  City)  are  likely  to  re¬ 
ceive  the  brunt  of  any  attack.  The  fact  that  the  perceived 
value  line  extends  beyond  the  true  value  line  for  index 
number  6  means  that  the  ship  will  be  targeted  with  more 
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Exhibit  13.  The  trade  off  map  for  threat  Sneaker  combining  several 

aspects  of  surveillance  effectiveness  and  information  denial. 
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threats  than  its  true  value  warrants.  The  reverse  situa¬ 
tion  is  indicated  for  the  index  number  1  ship.  The  cum¬ 
ulative  detection  probability  contours  are  given  so  that  an 
EW  officer  can  see  how  well  the  heavily  targeted  ships 
are  protected. 

The  second  trade  off  display  is  also  intended  to 
allow  joint  evaluation  of  divergent  aspects  of  surveil¬ 
lance  effectiveness  and  information  denial  but  it  also 
acts  as  a  bridge  for  comparisons  with  other  candidate 
EMCON  plans.  The  data  presented  in  this  display  (see 
Exhibit  14)  are  the  surveillance  scores  (the  interception 
probabilities)  and  the  information  denial  scores.  On  the 
left  ordinate,  the  surveillance  scores  are  plotted  and  on 
the  right  the  information  denial  scores  are  plotted.  On 
the  abscissa,  the  weights  that  could  be  assigned  to  infor¬ 
mation  denial  are  plotted.  On  the  far  right,  the  list  of 
EMCON  plans  for  which  the  scores  have  been  plotted 
appears.  The  EMCON  plans  are  ranked  in  terms  of  the 
information  denial  score  (the  EW).  The  lowest  EW 
score  is  from  plan  ALL  ON  which  also  has  the  highest 
surveillance  score.  The  data  for  this  plan  are  plotted 
on  the  graph  with  the  surveillance  score  of  .  68  and  the 
information  score  of  .  16  and  with  a  line  connecting  these 
two  points.  The  scores  for  each  other  plan  are  plotted 
in  a  similar  manner. 

This  display  facilitates  comparisons  across  plans. 

If  the  only  important  need  of  an  EMCON  plan  is  surveil¬ 
lance  coverage,  then  the  plan  corresponding  to  the  line 
intersecting  the  surveillance  ordinate  at  the  highest 
point  should  be  chosen  (ALL  ON).  If  information  denial 
is  of  paramount  importance,  then  the  plan  represented 
by  the  line  intersecting  the  information  denial  ordinate 
at  the  highest  point  should  be  chosen  (INFO).  If  both 
the  information  denial  and  surveillance  coverage  aspects 
are  equally  important,  then  the  plan  corresponding  to  the 
highest  line  above  the  point  at  which  both  are  given  equal 
weight  should  be  chosen  (SPRUON). 

The  final  display  attempts  to  carry  the  ideas  of 
surveillance  effectiveness  and  information  denial  one 
step  further  and  to  test  a  plan.  The  test  is  based  on  a 
simulated  strike  against  the  task  force  in  which  threats 
are  allocated  in  proportion  to  perceived  value  and  the 
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Exhibit  14.  Trade  off  display  combining  surveillance  and  information  denial 
scores  from  selected  EMCON  plans  for  purposes  of  comparison 
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threats  are  destroyed  according  to  the  interception  prob¬ 
abilities.  This  display,  the  trade  off  strike  display,  is 
presented  in  Exhibit  15. 

The  information  supplied  in  the  trade  off  strike  dis¬ 
play  includes  the  fractional  value  of  the  task  force  remain¬ 
ing  after  the  strike.  This  figure  is  to  some  extent  intended 
as  an  overall  measure  of  the  goodness  of  an  EMCON  plan. 
Generally,  the  higher  the  fractional  value  remaining,  the 
better  the  plan.  Other  information  supplied  concerns  the 
effect  of  the  strike  on  each  ship:  (1)  the  threat  allocation, 
(2)  the  penetration  probability,  (3)  the  number  of  hits,  (4) 
the  fractional  value  of  the  ship  remaining,  (5)  the  initial 
ship  value,  and  (6)  the  value  remaining. 
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Exhibit  15.  The  trade  off  strike  display 
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II.  EVALUATION  PURPOSE  AND  SPECIFIC  APPROACH 


The  purposes  of  the  current  evaluation  included  determining  the 
usefulness,  if  any,  of  the  EWAR  aid  in  deriving  EMCON  plans.  Specif¬ 
ically,  the  evaluation  was  designed  to  test  four  hypotheses; 

(1)  EMCON  plans  produced  using  the  decision  aid  will  be 
superior  to  plans  produced  without  the  aid 

(2)  EMCON  plan  training  will  further  enhance  the  quality 
of  the  EMCON  plans 

(3)  The  difficulty  of  the  scenario  problems  will  not  affect 
the  quality  of  the  EMCON  plans  generated  in  the  aided 
condition 

(4)  There  will  be  a  high  correlation  between  constructed 
problem  difficulty  and  perceived  problem  difficulty. 

The  hypotheses  and  evaluative  design  were  derived  as  the  result  of 
an  interested  parties  meeting  attended  by  members  of  the  staffs  of  Applied 
Psychological  Services,  Decision  Science  Applications,  Office  of  Naval  Re¬ 
search,  and  the  Department  of  Decision  Sciences,  University  of  Pennsylvania. 

The  first  hypothesis  was  tested  by  using  a  within  groups  design  in  which 
each  participant  was  required  to  develop  EMCON  plans  both  with  and  without 
the  use  of  the  EWAR  aid. 

The  second  hypothesis  was  tested  by  means  of  a  between  groups  design 
in  which  one  group  received  extensive  training  in  EMCON  planning  while  a 
second  group  received  no  training. 

The  third  hypothesis  was  tested  by  employing  problems  which  varied 
along  a  dimension  of  difficulty. 

The  final  hypothesis  was  tested  by  comparing  judgments  made  by  par¬ 
ticipants  about  the  difficulty  of  each  scenario  with  the  a  priori  difficulty  of 
the  problems. 

In  essence,  a  basic  two-by-two-by-two  split  plot  factorial  design 
with  block-treatment  confounding  was  employed.  The  first  factor  was  the 
between  subjects  (blocks)  training  factor,  the  second  factor  was  the  within 
subjects  aiding  factor,  and  the  third  factor  was  the  within  a  within  block 
difficulty  factor.  A  representation  of  the  basic  design  is  presented  in 
Exhibit  16. 
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Independent  Variables 


Training 

The  training  was  concerned  with;  (1)  the  broad  aspects  of  EMCON 
planning,  (2)  a  set  of  manual  methodologies  for  evaluating  plans,  and  (3) 
information  denial  rules  (Exhibit  17)  and  their  applications.  To  assess  the 
effects  of  the  training,  two  groups  of  subjects  were  used.  One  may  be 
considered  to  represent  an  experimental  group  and  the  second  gr  oup  may 
be  considered  to  represent  a  control  group.  Because  of  the  impossibility 
of  separating  the  training  in  manual  EMCON  planning  from  the  training  in 
the  information  denial  guidelines,  these  two  aspects  were  combined  in  the 
training  received  by  the  experimental  group. 

The  manual  techniques  included  in  the  training  could  be  used  to  es¬ 
timate,  with  the  use  of  a  hand  calculator,  most  of  the  indices  supplied  by  the 
aid,  e.  g.  ,  the  surveillance  circles,  cumulative  probabilities,  information 
denial  details,  trade  offs,  and  the  like. 

The  information  denial  guidelines  were  six  rules  which  could  be  applied 
to  a  task  force  to  minimize  information  given  away.  The  guidelines  were 
statements  of  what  emitters  should  be  activated  or  deactivated  in  order  to 
maximize  information  denial.  The  first  three  suggested  how  to  deny  targeting 
information  while  the  last  three  suggested  means  whereby  a  high  value  ship 
could  be  "hidden.  "  Application  of  the  guidelines  to  a  scenario  resulted  in  an 
EMCON  plan  which  maximally  denied  information  to  an  enemy. 

Aid  Levels 


The  main  test  of  the  effectiveness  of  the  EWAR  decision  aid  involved 
comparing  EMCON  plans  developed  with  the  aid  with  plans  developed  man¬ 
ually.  Therefore,  there  were  two  aid  levels;  (1)  a  full  aid  condition  and  (2) 
a  no  aid  condition.  Each  participant  was  exposed  to  both  the  full  aid  and  the 
no  aid  conditions.  In  the  full  aid  condition,  all  of  the  displays  and  aid  gen¬ 
erated  information,  described  previously,  were  available  to  the  subjects. 

In  the  no  aid  condition,  only  a  very  restricted  set  of  information  was  made 
available.  However,  additional  information  could  be  generated  through  hand 
computational  methods,  at  the  option  of  the  individual  participant.  The  infor¬ 
mation  made  available  in  the  no  aid  condition  included  radar  order-of -battle 
information  and  the  surveillance  range  information.  The  information  supplied 
in  the  no  aid  condition  was  chosen  to  reflect  the  type  of  information  an  EW 
officer  would  be  likely  to  have  immediately  available  under  normal  operating 
conditions. 
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Information  Denial  Guidelines 


1.  Avoid  turning  on  a  unique  radar  or  a  unique  group  of 
radars. 

2.  Turn  on  all  radars  common  to  all  ships. 

3.  Look  for  radar  groups;  turn  such  radars  off  or  on 
as  a  group. 

4.  Turn  off  all  radars  in  the  task  force  that  are  not 
aboard  a  ship  to  be  "hidden.  " 

5.  Turn  off  aLl  radars  aboard  any  ship  to  be  hidden, 
except  for  those  radars  which  are  common  to  all 
ships. 

G .  Turn  on  a  radar  or  radars  aboard  an  intermediate 
value  ship  where  such  an  action  would  result  in  an 
intermediate  value  ship  being  identified  as  a  high 
value  ship  and  the  high  value  ship  being  identified 
as  a  low  value  ship. 


Exhibit  17.  The  information  denial  guidelines  used  in  the  training 
of  the  experimental  group. 
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Difficulty 

It  was  assumed  that  the  difficulty  of  developing  an  EMCON  plan 
would  be  a  function  of  a  number  of  variables.  Difficulty  was  assumed  to  be 
positively  related  to  three  factors;  (1)  the  amount  of  information  pro¬ 
cessing  inherent  in  a  problem,  (2)  threat  heterogeneity,  and  (3)  number  of 
ships  in  the  task  force. 

The  magnitude  of  the  information  processing  in  developing  an  EMCON 
plan  relates  strongly  to  two  considerations;  (1)  differences  in  the  radars 
across  ships,  and  (2)  differences  in  ships  across  radars.  Each  of  these  is 
related  to  cues  that  could  be  used  to  identify  ships.  As  the  number  of  such 
cues  increases,  so  would  the  magnitude  of  the  information  processing  load. 
Accordingly,  information  load  may  be  viewed  as  a  measure  of  the  difficulty 
of  formulating  an  EMCON  plan  for  a  given  situation.  To  facilitate  the  as¬ 
sessment  of  the  information  processing  load,  a  complication  score  was  ob¬ 
tained  for  each  problem  included  within  the  present  evaluation.  The  compli¬ 
cation  score  was  the  product  of  the  sum  of  the  differences  of  the  radars  across 
ships  and  the  sum  of  the  differences  in  ships  across  radars. 

The  complication  score  was  used  along  with  the  other  two  factors  to 
produce  a  difficulty  score  for  each  problem.  The  number  of  different  poten¬ 
tial  threats  was  expected  to  increase  difficulty  because  an  EMCON  plan  suited 
for  one  threat  type  might  be  inadequate  against  another  threat  type. 

It  was  also  assumed  that  the  number  of  ships  to  be  protected  affects 
the  difficulty  of  a  problem.  These  three  factors;  complication  score,  he¬ 
terogeneity  of  threats,  and  number  of  ships,  were  weighted  and  summed  to 
produce  a  measure  of  difficulty  for  each  problem; 

Difficulty  scores  =  CS  +  wiT  +  W2S 

where  CS  =  complication  score 

T  *  number  of  different  threats 
S  =  number  of  ships  in  the  task  force 
w  1  -  50 
W2  =  20 

The  weights,  wj  and  W2,  were  incorporated  to  make  the  variance  of 
the  final  two  factors  approximate  the  variance  of  the  first  factor. 

To  assess  the  effects  of  difficulty  on  EMCON  planning,  problems  were 
chosen  so  that  half  of  the  problems  had  relatively  low  difficulty  scores  and 
half  had  relatively  high  difficulty  scores. 
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Dependent  Variables 

Quantitative 

The  major  dependent  measure  was  based  on  the  EMCON  plan  each 
participant  developed  as  his  best  solution  for  each  problem.  The  meas¬ 
ure  of  the  goodness  of  each  of  the  participants’  chosen  plans  was  the  frac¬ 
tional  value  of  the  task  force  remaining  after  a  strike  when  the  task  force 
used  the  plan.  This  measure  was  produced  by  the  aid  and  was  available  to 
the  participants  in  the  aided  condition.  The  measure  assesses  three  different 
aspects  of  an  EMCON  plan--the  information  denial,  the  surveillance  effective¬ 
ness,  and  the  trade  off  between  them. 

Criterion 

The  dependent  measure,  fractional  value  remaining,  could  be  com¬ 
pared  both  between  and  within  conditions.  However,  it  was  also  thought  that 
each  of  the  participants’  EMCON  plans  should  be  compared  with  a  criterion 
representing  the  best  possible  plan.  To  develop  the  criterion  for  each  prob¬ 
lem,  the  EMCON  plan  in  which  the  fractional  value  remaining  was  highest  was 
constructed  and  the  fractional  value  of  the  task  force  remaining  under  each  of 
these  plans  was  used  as  a  measure  for  evaluating  the  goodness  of  the  experi¬ 
mental  subjects'  derived  plans.  Difference  scores  were  developed  and  were 
used  to  estimate  how  closely  the  participants'  plans  approximated  the  best  pos¬ 
sible  plans  and  how  they  varied  with  the  independent  variables. 

Difficulty  and  Confidence  Ratings 

In  addition,  the  participants  were  asked  to  rate  each  problem  on  two 
dimensions.  They  were  first  asked  to  indicate  how  confident  they  were  in  each 
solution  and,  second,  how  difficult  it  was  to  arrive  at  the  solution.  Magnitude 
estimation  techniques  were  used  to  obtain  both  judgments.  The  difficulty 
judgments  were  used  to  analyze  the  relationship  between  perceived  difficulty 
and  the  a  priori  difficulty  scores  which  were  calculated  as  described  above. 

In  addition,  there  was  also  interest  in  the  relationship  between  perceived  dif¬ 
ficulty  and  confidence. 

Percentage  Correct 

An  EMCON  plan  could  be  thought  of  as  a  set  of  independent  statements 
about  the  status  of  each  of  the  radar  emitters  in  the  task  force.  Using  this 
conceptualization,  another  way  to  compare  the  criterion  plans  with  the  plans 
of  the  participants  becomes  obvious.  Participants'  plans  could  be  graded  on 
how  many  emitters  that  should  be  on,  according  to  the  criterion  plan,  were 
actually  on,  and  how  many  emitters  that  should  be  off,  were  off.  A  per¬ 
centage  correct  score  could  then  be  derived.  In  addition,  the  data  could  be 


assessed  to  determine  how  the  distributions  of  false  positives  and  negatives 
vary  across  and  within  conditions  and  how  they  may  relate  to  each  other. 


Qualitative 

After  each  participant  completed  the  last  of  the  test  scenario  problems, 
he  was  interviewed  on  a  variety  of  topics.  Generally,  three  types  of  informa¬ 
tion  were  solicited:  (1)  categorical  ratings  and  rank  orderings  of  the  useful¬ 
ness  of  the  aid,  (2)  descriptions  of  strategies  and  heuristics  used  to  develop 
and  evaluate  EMCON  plans,  and  (3)  estimates  of  the  relationship  between 
the  aid  and  its  goals.  The  data  on  how  closely  the  EWAR  aid  approximated 
its  goals  provided  the  basis  for  a  multiple  attribute  utility  analysis.  To 
obtain  the  goals  and  their  weights  as  required  for  the  analysis.  Decision 
Science  Applications  supplied  seven  goals,  or  general  objectives  of  the 
EWAR  aid.  They  ranked  these  goals  from  most  important  to  least  impor¬ 
tant  and  weighted  them  as  to  their  importance  by  distributing  100  points 
among  them.  The  goals,  ranked  and  weighted,  are  presented  in  Exhibit  18. 
The  full  interview  content  is  presented  in  Appendix  A. 


GoaL 


Weight 


Gl.  To  facilitate  the  formation  of  hypotheses  about  how 

best  to  improve  a  current  plan  10 

G2.  To  produce  better  EMCON  plans  20 

G3,  To  help  quantify  the  surveillance  and  information 

consequences  of  a  plan  15 

G4.  To  make  it  practical  to  examine  a  larger  set  of 

alternative  plans  10 

G5.  To  understand  how  different  factors  (threats,  ra¬ 
dars,  etc.)  contribute  to  EMCON  planning  15 

G6.  To  improve  planner  confidence  in  the  quality  of  his 

EMCON  plans  15 

G7.  To  facilitate  comparison  of  alternative  plans  15 


Exhibit  18.  Goals  of  EWAR  decision  aid  and  their  weights. 
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Subjects,  Methods,  and  Procedures 

Subjects 

The  participants  were  16  members  of  the  faculty  of  the  United  States 
Naval  Academy  who  volunteered  to  participate  in  the  EWAR  evaluation. 
Thirteen  were  Naval  officers  and  three  were  Marine  officers.  Of  the  Naval 
officers,  five  were  Lieutenants;  six  were  Lieutenant  Commanders;  one  was 
a  Commander,  and  one  was  a  Captain.  The  Marine  officers  were  all  Cap¬ 
tains.  The  average  time  in  the  service  of  the  total  group  was  about  12  years. 


Apparatus 

The  evaluation  was  conducted  at  the  facilities  of  the  Decision  Aiding 
Systems  Laboratory,  Department  of  Decision  Science,  Wharton  School, 
University  of  Pennsylvania.  The  EWAR  data  acquision  program  operated 
on  line  with  a  DEC- 10  computer  (located  in  the  Wharton  computer  facility) 
in  a  time  sharing  mode  during  the  aided  condition.  In  addition,  the  DEC- 10 
operated  a  variety  of  peripheral  equipments  in  the  laboratory  proper. 

The  arrangement  of  the  evaluation  equipment  and  the  general  organ¬ 
ization  of  the  laboratory  room  are  presented  schematically  in  Figure  1.  The 
laboratory  room  was  partitioned  into  two  areas --an  experimental  area  and 
a  support  area.  The  experimental  area  was  in  turn  divided  into  two  areas, 
area  "A"  and  area  "B",  each  of  which  was  designed  for  the  use  of  one  par¬ 
ticipant.  At  one  end  of  the  experimental  section,  two  stations,  which  were 
separated  by  a  partition,  were  constructed  to  be  used  by  the  participants  in 
the  aided  condition.  These  sections  will  be  called  EWAR  stations.  Two  ad¬ 
ditional  sections,  used  by  the  participants  in  the  no  aid  condition- -the  man¬ 
ual  stations--were  arranged  at  the  other  end  of  the  room. 

At  each  EWAR  station,  there  was  a  Concept  100  terminal  and  a  Tek¬ 
tronix  4013  display.  Commands  were  entered  through  the  Concept  100  key¬ 
board  and  alphanumeric  displays  were  presented  on  its  cathode  ray  tube. 
Graphic  displays  were  presented  on  the  Tektronix  4013.  Two  devices,  cap¬ 
able  of  producing  hard  copies,  were  shared  by  the  participants.  One,  the 
Tektronix  4010,  was  situated  in  front  of  the  partition  separating  EWAR 
station  "A"  from  station  "B.  "  It  produced,  on  participant  request,  a  copy 
of  any  of  the  graphics  displayed  on  the  Tektronix  4013.  The  other,  a  DEC- 
Writer,  produced  on  demand  hard  copies  of  the  alphanumeric  displays  pre¬ 
sented  on  the  Concept  100.  The  DEC-Writer  was  located  adjacent  to  the 
Tektronix  4010  copying  machine. 

The  manual  stations  consisted  of  a  table  and  chair  for  each  partic¬ 
ipant  along  with  any  necessary  materials,  such  as  a  hand  calculator,  scratch 
paper,  pencils,  and  overlays  representing  maximum  radar  range  circles. 


INTERVIEW  AREA 


"A" 

EWAR  STATION 


EWAR  STATION 


MANUAL  STATION 


- - \  MAN 

chain  j  ^ 


MANUAL  STATION 


TABLE 


□  opapp 


MAXIMUM  RANGE  OVERLAYS 


Figure  1  Schemotic  of  equipment  orrongement  for  the  EWAR  evoluotion. 


The  overlays  provided  a  convenient  technique  for  hand  production  and 
visualization  of  maximum  range  circles.  Such  production  may  have  been 
time  consuming  with  a  hand  drawing  compass. 


Group  Assignments  and  Prerequisite  Training 

The  participants  were  divided  into  two  groups:  an  experimental  and 
a  control  group.  Eight  persons  were  assigned  randomly  to  each  group. 

The  experimental  group  was  given  one  and  half  days  of  prerequi¬ 
site  training  at  the  Naval  Academy.  The  first  one  and  a  quarter  days  of 
this  training  were  devoted  to  training  the  experimental  group  in;  (1)  the 
problems  of  emission  control,  (2)  the  manual  development  and  testing  of 
EMCON  plans,  and  (3)  the  applications  of  the  information  denial  guidelines. 
The  training  was  conducted  using  an  audio-visual-lecture  technique  and 
strictly  followed  lesson  plans  jointly  developed  by  Decision  Science  Ap¬ 
plications  and  Applied  Psychological  Services.  Within  the  lesson  plans, 
there  was  considerable  emphasis  on  practice  exercises  within  three  typical 
scenario  problems,  and  on  the  development  and  evaluation  of  a  number  of 
EMCON  plans  for  each  problem. 

The  last  one  quarter  day  of  training  was  devoted  to  the  EWAR  de¬ 
cision  aid  and,  for  this  period,  the  experimental  and  the  control  groups 
were  both  present.  Before  the  control  group  was  permitted  into  the  class¬ 
room,  the  experimental  group  was  advised  about  the  necessity  for  keeping 
the  contents  of  the  earlier  training  secret  and  the  group  was  asked  to  re¬ 
frain  from  discussing  the  content  with  the  members  of  the  control  group. 

In  addition,  the  experimental  group  was  advised  not  to  ask  any  questions 
which  would  relate  their  training  to  the  EWAR  aid's  displays. 

The  information  about  the  EWAR  discussed  in  the  joint  briefing 
consisted  of  a  full  description  and  interpretation  of  the  displays  made  a- 
vailable  by  the  aid.  It  was  made  clear  to  the  participants  that  the  pur¬ 
pose  of  the  briefing  was  to  provide  familiarization  with  the  aid  and  that 
additional  training  on  how  to  use  the  aid  would  ultimately  be  given  at  the 
University  of  Pennsylvania  when  they  arrived  to  participate  in  the  actual 
evaluation. 

Problems 


Decision  Science  Applications  created  15  EMCON  problems,  three 
of  which  were  used  during  the  training  at  the  Academy.  From  the  remain¬ 
ing  12,  eight  were  used  in  testing  and  two  were  used  as  practice  problems. 
The  eight  test  problems  wer  e  selected  on  the  basis  of  their  difficulty  scores 
(described  earlier)  and  classified  as  either  "easy"  or  "hard."  The  four  as¬ 
signed  to  the  "easy"  category  had  a  mean  difficulty  rating  of  216  with  a 
range  of  160  to  242.  The  four  categorized  as  "har'd"  had  a  mean  difficulty 
rating  of  379  and  a  range  of  320  to  409. 


Problem  Presentation  Sequence 


The  sequence  of  problem  presentation  for  each  participant  in  the 
experimental  group  is  shown  in  Figure  2.  The  same  sequence  was 
followed  for  the  control  group.  Each  participant  was  required  to  work 
through  eight  problems- -four  aided,  four  unaided--four  of  which  were 
"easy"  and  four  of  which  were  "hard."  Each  participant  was  exposed  to 
the  aid  levels  in  a  double  rotated  ABBA  sequence  in  order  to  distribute 
the  more  pervasive  sequential  effects  evenly  across  aid  levels.  Across 
the  ABBA  distribution,  problem  difficulty  was  alternated. 

The  ordering,  shown  in  Figure  2,  was  produced  by  first  randomly 
assigning  symbols  Ei,  E2,  E3,  E4,  and  Hi,  H2,  H3,  H4,  to  the  "easy" 
and  to  the  "hard"  problems,  respectively.  These  problems  were  then 
ordered,  El,  Hi,  .  .  .  ,  E4,  H4,  to  produce  a  sequence  of  successively 
alternating  "easy"  and  "hard"  problems  to  be  used  as  the  basic  ordering 
of  problems  for  the  evaluation. 

This  order  was  received  by  the  first  participant,  (Sj,  in  Figure  2) 
starting  with  the  aided  condition  and  by  the  third  subject,  S3,  starting  with 
the  unaided  condition.  Both  S2  and  S4  received  this  sequence  in  reverse 
order  but  each  problem  was  aided  or  unaided  in  the  same  manner  as  for 
Si  and  S3,  respectively.  The  same  sequence  was  split  and  reversed  from 
the  middle  for  S5  and  S7  and  the  split  and  reversed  sequence  were  com¬ 
pletely  reversed  for  Sq  and  Sg.  S5  and  S6  began  with  an  aided  condition  and 
S7  and  Sg  began  with  an  unaided  condition.  This  presentation  sequence  allo¬ 
cated  each  problem  equally  often  to  both  the  aided  and  the  unaided  conditions. 
The  design  did  not  fully  control  for  serial  effects  but  to  do  this  completely 
would  require  an  unwarranted  increase  in  the  number  of  subjects.  However, 
the  design  controlled  for  order  and  sequence  of  presentation  so  that  a  rather 
detailed  analysis  of  any  learning  effects  was  possible. 


Each  subject  in  the  control  group  received  the  same  problems  in 
the  same  order  as  the  same  numbered  subject  in  the  experimental  group. 

Data  Collection 


Each  participant  was  allocated  between  one  and  a  half  and  two 
hours  for  each  problem.  The  time  allocation  for  each  problem  depended 
on  the  problem's  rated  difficulty.  At  the  end  of  the  allocated  time,  or 
when  a  participant  indicated  that  he  had  decided  on  a  best  EMCON  plan 
for  the  problem,  he  was  asked  to  supply  three  sets  of  information:  (1) 
the  best  EMCON  plan,  (2)  a  difficulty  rating  tor  the  problem,  and  (3) 
a  confidence  rating. 

The  EMCON  plan  was  recorded  by  simply  listing  the  ships  in  the 
task  force  that  had  activated  radar  emitters  and  by  marking  the  box  cor¬ 
responding  to  the  radars  emitting  on  each  ship.  With  this  information. 
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Sequential  Presentation  of  Aid  and  Difficulty  Levels 


A 

B 

B 

A 
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A 
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A: 

Aided 

H:  Hard  problems 

B: 

Unaided 

E:  Easy  problems 

Figure  2.  Schedule  of  four  easy  (E)  and  four  hard  (H)  problems 
distributed  across  coun.terbaLan.ced  ABBA  sequences 
for  the  experimental  group. 
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the  rest  of  the  EMCON  plan  was  directly  deduced  and  the  relevant  value  of 
each  dependent  measure  could  be  generated.  Each  participant  in  the  eval¬ 
uation  was  exposed  to  two  practice  problems. 

Procedure- -Specific  Practice  and  Testing 

Each  participant  was  allocated  a  total  of  16  hours.  The  time  was 
distributed  over  two  successive  days  of  eight  hours  each.  The  first  three 
hours  were  devoted  to  tutoring  the  participant  in  the  use  of  the  EWAR  de¬ 
cision  aid  and  the  other  13  hours  were  devoted  to  the  actual  evaluation  of  the 
aid.  The  participants  were  scheduled  for  the  evaluation  in  pairs.  Both 
members  of  a  pair  were  assigned  to  the  same  group,  either  experimental 
or  control.  This  procedure  was  instituted  so  that  the  members  of  the  ex¬ 
perimental  group  could  receive  a  review  of  the  use  of  the  information  de¬ 
nial  guidelines  which  formed  a  part  of  their  earlier  training. 

When  the  participants  arrived  at  the  laboratory,  the  purpose  of  the 
evaluation  was  rearticulated,  and  the  importance  and  possible  contributions 
that  their  time  and  effort  would  make  to  the  project  was  stressed.  Following 
this,  the  schedule  for  the  next  two  days  was  outlined.  Then,  the  practice 
session  began  and  the  participants  and  the  evaluator  worked  together. 

To  facilitate  interaction  with  the  EWAR  aid,  each  participant  was 
given  a  list  of  the  commands  with  which  he  could  modify,  save,  or  retrieve 
EMCON  plans,  as  well  as  call  up  desired  displays.  (These  commands  are 
presented  in  Appendix  B. )  Then,  the  evaluator  acted  as  the  system  operator 
and,  in  conjunction  with  the  participant,  worked  through  the  first  practice 
problem  calling  up  the  various  displays  in  the  order  in  which  they  were  listed 
in  the  handout  while  the  participant  watched.  The  content  of  each  display  was 
described  and  any  questions  were  answered.  Next,  the  evaluator  used  the  op¬ 
erational  commands  to  modify  the  EMCON  plan.  The  participant  was  queried 
as  to  how  the  radar  emission  configuration  should  be  altered.  If  the  partici¬ 
pant  was  a  member  of  the  experimental  (trained)  group,  the  evaluator  sug¬ 
gested  that  individual  radars  should  be  turned  on  or  off  to  satisfy  the  infor¬ 
mation  denial  guidelines.  However,  if  the  participant  was  from  the  control 
group,  then  the  choice  was  left  to  the  participant  and  the  evaluator  made  no 
suggestions.  After  modifying  the  EMCON  plan,  the  displays  were  again 
called  and  the  resultant  changes  were  indicated  and  explained.  There  was 
no  attempt  made  to  fully  develop  the  best  EMCON  plan  for  this  practice  prob¬ 
lem.  The  intent  was  to  familiarize  the  participant  with  the  use  of  the  EWAR 
aid. 


Then,  the  second  practice  problem  was  worked  through.  This  time, 
the  participant  acted  as  the  system  operator.  Before  starting,  the  fact  was 
explained  that  this  problem  would  be  used  as  the  modulus  for  the  difficulty 
and  confidence  ratings  of  the  test  problems.  A  clear  and  concise  statement 
concerning  the  magnitude  estimation  technique  was  made.  Both  difficulty 


(how  difficult  it  was  to  arrive  at  a  preferred  EMCON  plan)  and  confidence 
(how  confident  the  participant  was  in  the  plan)  were  assigned  the  same 
modulus  value  of  100.  Therefore,  if  the  first  test  problem  was  twice  as 
difficult  as  the  second  practice  problem,  then  the  participant  would  assign 
a  difficulty  rating  of  200  to  that  test  problem.  In  a  similar  vein,  if  the 
participant  was  half  as  confident  in  the  EMCON  plan  for  the  first  test  prob¬ 
lem  as  he  was  for  the  second  practice  problem,  he  would  be  expected  to 
assign  a  confidence  rating  of  50  to  the  first  test  problem. 

Data  recording  sheets  were  also  distributed  and  explained  at  this 

point. 

Once  he  understood  how  to  record  the  best  plans  and  the  ratings,  the 
participant  was  allowed  to  use  the  aid  to  work  through  the  second  practice 
problem.  The  evaluator  remained  present  and  helped,  as  required,  with 
the  mechanics  of  the  peripheral  hardware  and  the  command  structure. 

Again,  as  with  the  first  practice  problem,  the  evaluator  attempted,  if  nec¬ 
essary,  to  help  a  participant  in  the  experimental  group  in  the  applications 
of  the  information  denial  guidelines.  This  assistance  was  not  given  to  the 
control  subjects. 

About  20  minutes  before  the  expiration  of  the  allocated  time  for  the 
problem,  the  participant  was  told  that  the  allocated  time  had  just  about  ex¬ 
pired  and  that  he  should  try  to  make  a  decision.  When  the  time  expired,  or 
when  the  participant  indicated  that  he  had  made  a  decision,  the  evaluator  as¬ 
sured  that  the  plan  was  properly  recorded.  At  this  time,  the  significance  of 
the  moduli  and  how  to  use  them  were  reiterated. 

After  the  participant  completed  the  second  practice  problem,  he  was 
allowed  to  start  immediately  on  his  first  test  problem.  Each  participant 
was  assigned  a  subject  number  associated  with  the  preestablished  ordering 
of  the  problems  and  sequence  of  aid  levels. 

While  the  participant  worked  through  the  problems,  the  evaluator 
waited  in  the  support  area  and  remained  prepared  to  intervene  if  any  prob¬ 
lem  arose. 

As  a  participant  finished  each  problem,  the  evaluator  ascertained 
that  he  had  properly  completed  the  data  sheet.  Then,  the  handout  for  the 
next  problem  was  given  to  him  and,  after  being  told  whether  or  not  to  use 
the  aid,  the  participant  was  allowed  to  begin  work  on  the  problem. 

Post  Data  Collection  Interview 

After  completing  all  eight  problems,  each  participant  was  indi¬ 
vidually  interviewed.  The  interview  was  conducted  by  the  evaluator  in  the 
interview  area  and  consumed  about  one  hour. 
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The  EWAR  aid  evaluation  was  designed  to  provide  both  quantita¬ 
tive  and  qualitative  information  about  the  EWAR  aid  when  it  is  used  in  a 
laboratory  situation.  The  obtained  best  EMCON  plans  of  each  participant 
were  first  subjected  to  a  set  of  analyses  to  determine  the  effects  of  each  of 
the  independent  variables  (use  of  aid,  training,  and  problem  difficulty)  on 
the  plans  produced. 

Fraction  of  Own  Force  Remaining 

A  fractional  value  of  the  task  force  remaining  (FRAC)  score  was  de¬ 
rived  for  each  best  participant  plan  by  taking  the  mean  (as  computed  by  the 
aid)  of  the  fractional  value  remaining  after  two  strikes  against  the  task  force-- 
one  by  Sneaker  and  the  other  by  Streaker.  The  FRAC  score  represented  a 
trade  off  between  surveillance  coverage  and  information  denial.  Indeed,  to 
obtain  the  highest  possible  FRAC  score,  it  was  necessary  to  consider  both 
surveillance  protection  and  information  denial.  This  suggests  that  the  FRAC 
score  was  a  quite  adequate  measure  of  the  overall  quality  of  an  EMCON  plan. 
Moreover,  this  score  is  essentially  a  "payoff"  score  and  may  be  considered 
to  reflect  directly  overall  plan  effectiveness. 

The  distribution  of  the  FRAC  scores  was  examined  through  a  two 
(training-no  training)  by  two  (aided -unaided)  by  two  ("easy"-"hard")  analysis 
of  variance.  Within  each  training  by  aid  level  by  difficulty  cell,  each  sub¬ 
ject  was  observed  twice.  The  data  used  in  this  variance  analysis  and  all 
those  reported  herein  were  the  mean  of  the  two  observations.  The  results  of 
the  variance  analysis  are  summarized  in  Table  1.  The  variance  produced  by 
the  aid  levels  was  statistically  significant.  No  statistically  significant  vari¬ 
ance  could  be  attributed  to  the  other  main  effects.  The  mean  FRAC  scores 
expressed  in  standard  deviates  for  each  condition  were: 


Training  Levels  Mean 


Experimental 

-.07 

Control 

.07 

Aid  Levels 

Aided 

1.  44 

Unaided 

-1. 44 

Difficulty  Levels 

"Easy" 

-,3b 

"Hard" 

.  36 

The  mean  FRAC  scores  were  not  affected  by  the  training  or  by  the 
problem  difficulty  but  they  were  clearly  influenced  by  the  use  of  the  aid. 
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Table  1 


Variance  Analysis  Summary  for  the  FRAC  Scores 


Sum  of 


Source 

Squares 

Between  Groups 

6.554 

Training  (T) 

0.009 

Subjects  Within 

Groups 

6.545 

Within  Conditions 

21.010 

Aid  Use  (A) 

3.530 

T  x  A 

0.044 

A  x  Subjects  Within 

Groups 

3.530 

Difficulty  (D) 

0.224 

T  x  D 

0.132 

D  x  Subjects  Within 

Groups 

7.394 

A  x  D 

0.274 

T  x  A  x  D 

0.454 

A  x  D  x  Subj  ects 

Within  Groups 

5.768 

Total 

27.56 

Mean 

d.f. 

Square 

F 

15 


1 

0.009 

0.019 

14 

0.468 

48 

1 

3.530 

14.010* 

1 

0.044 

0.175 

14 

0.252 

1 

0.224 

0.424 

1 

0.132 

0.250 

14 

0.528 

1 

0.274 

0.665 

1 

0.454 

1.100 

14 

0.412 

63 


*p  »  <  . 01 
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Moreover,  the  variance  accounted  for  by  ail  the  independent 
variables  was  about  the  same  as  the  variance  accounted  for  by  the  aid 
levels  alone.  It  appears  that  for  the  proportion  of  the  variance  accounted 
for,  that  proportion  can  be  understood  to  be  largely  a  function  of  only 
the  aid  levels. 

Comparison  With  Criterion  Scores 

To  examine  more  fully  the  best  EMCON  plans  of  the  participants 
in  the  EWAR  evaluation,  the  participants'  FRAC  scores  were  compared 
with  a  criterion  score. 

The  criterion  fraction  (CFRAC)  scores  were  calculated  by  gen¬ 
erating  the  best  EMCON  plan  for  each  problem.  The  best  EMCON  plan 
was,  by  definition,  the  one  associated  with  the  highest  fractional  value 
remaining,  averaged  across  both  threat  types. 

The  relationship  between  the  criterion  and  the  participants'  EMCON 
plans  was  studied  by  analyzing  the  difference  (DFRAC)  between  the  CFRAC 
and  FRAC  scores.  The  obtained  DFRAC  scores  varied  from  zero,  where 
the  FRAC  and  DFRAC  were  equal,  to  3.  62. 

The  DFRAC  scores  were  subjected  to  a  two-by-two-by-two  analysis  of 
variance.  The  results  are  summarized  in  Table  2.  Again,  a  statistically  sig¬ 
nificant  effect  was  attributed  to  the  manipulation  of  the  aid  levels.  The  mean 
difference  value  in  standard  score  form  for  each  condition  was; 


Training  Levels 

Mean 

Experimental 

-.42 

Control 

.34 

Aid  Levels 

Aided 

-1.  46 

Unaided 

1.61 

Difficulty  Levels 

"Easy" 

-.09 

"Hard" 

.  02 

Only  negligible  variance  was  attributable  to  the  manipulation  of 
the  training  or  to  the  problem  difficulty.  In  the  aided  condition,  the  DFRAC 
scores  tended  to  be  relatively  low,  with  a  mean  of  -1.46,  and  in  the  manual 
condition  they  tended  to  be  relatively  high,  with  a  mean  of  1.61.  Again, 
of  the  variance  accounted  for,  was  mostly  a  function  of  aid  levels. 


t 


43 


Table  2 


f 


Variance  Analysis  Summary  for  the  DFRAC  Scores 


Sum  of 

Mean 

Source 

Squares 

d.f . 

Square 

F 

Between  Groups 

3.929 

15 

Training  (T) 

Subjects  Within 

0.627 

1 

0.627 

2.658 

Groups 

3.302 

14 

0.236 

Within  Conditions 

25.208 

48 

Aid  Use  (A) 

8.960 

1 

8.960 

56.837* 

T  x  A 

A  x  Subjects  Within 

0.674 

1 

0.674 

2.858 

Groups 

2.207 

14 

0.158 

Difficulty  (D) 

0.018 

1 

0.018 

0.028 

T  x  D 

D  x  Subjects  Within 

0.014 

1 

0.014 

0.021 

Groups 

9.170 

14 

0.655 

A  x  D 

0.007 

1 

0.007 

0.029 

T  x  A  x  D 

A  x  D  x  Subjects 

0.815 

1 

0.815 

3.448 

Within  Groups 

3.309 

14 

0.236 

Total 

29.137 

63 

*P  =  <  0.01 
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It  appears  that  the  quality  of  the  EMCON  plans,  measured  in 
terms  of  either  FRAC  or  DFRAC  scores,  was  directly  related  to  whether 
or  not  the  participants  used  the  EWAR  aid.  The  other  independent  var¬ 
iables  manipulated  had  no  systematic  effects  on  either  measure. 

Percentage  Correct 

An  EMCON  plan  may  be  considered  as  a  set  of  statements  about 
the  status  of  emitters.  Within  this  logic,  a  criterion  EMCON  plan  could 
be  considered  to  be  a  statement  of  the  status  of  the  emitters  for  a  best 
plan.  Such  a  logic  considers  the  status  of  each  emitter  as  a  true-false 
item  and  allows  direct  scoring  of  each  participant 's  best  plan  for  each 
problem. 

Taking  the  proportion  of  the  total  number  of  emitters  correctly 
set  against  the  total  number  of  emitters  in  the  task  force  gave  a  percentage 
correct  (PERCOR)  measure  of  the  correspondence  between  the  criterion 
plans  and  the  participants'  plans.  Over  all  conditions,  the  mean  PERCOR, 
score  was  76.  80  with  a  standard  deviation  of  20.04.  These  scores  suggest 
that  about  70  percent  of  the  time  there  was  a  55  to  95  percent  correspond¬ 
ence  between  the  criterion  and  participant  generated  EMCON  plans. 

However,  it  was  possible  to  partition  the  PERCOR  score  into  two 
parts:  (1)  the  percentage  of  the  radars  correctly  turned  on  (PERON),  and 
(2)  the  percentage  of  the  radar  emitters  correctly  turned  off  (PEROFF). 

The  mean  PERON  score,  66.  71,  was  substantially  lower  than  the  mean 
PERCOR  score,  whereas  the  mean  PEROFF  score,  91.05  was  consider¬ 
ably  higher. 

The  t_-test  was  used  to  examine  the  difference  between  the  PERON 
and  PEROFF  scores.  A  statistically  significant  difference  was  observed. 
This  finding  suggests,  in  one  sense,  a  tendency  on  the  part  of  the  par¬ 
ticipants  to  underemphasize  surveillance  in  their  EMCON  planning. 

The  effects  of  the  manipulation  of  the  independent  variables  on 
the  PERCOR  scores  were  also  examined  by  means  of  a  two-by-two-by- 
two  analysis  of  variance.  The  results  of  the  variance  analysis  are  sum¬ 
marized  in  Table  3.  The  results  indicated  that  the  aid  level  manipulation 
contributed  significantly  to  the  PERCOR.  In  addition,  as  was  indicated 
previously  for  other  measures,  the  manipulation  of  training  and  of  prob¬ 
lem  difficulty  did  not  produce  any  statistically  significant  effects  on  the 
PERCOR  scores.  The  mean  PERCOR  scores,  across  and  within  conditions, 
were: 


Training  Levels  Mean 

Experimental  76.78 

Control  76.82 
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Table  3 


Variance  Analysis  Summary  for  the  PERCOR  Scores 


Sum  of 

Mean 

Source 

Squares 

d.f. 

Square 

F 

Between  Groups 

2,579.938 

15 

Training  (T) 

0.063 

1 

0.063 

.0003 

Subjects  Within 

Groups 

2,579.875 

14 

184.277 

Within  Conditions 

9,692.000 

48 

Aid  Use  (A) 

1,640.250 

1 

1,640.250 

8.5920* 

T  x  A 

0.025 

1 

0.025 

.0001 

A  x  Subjects  Within 

Groups 

2,672.725 

14 

190.909 

Difficulty  (D) 

80.999 

1 

80.999 

.6029 

T  x  D 

225.000 

1 

225.000 

1.675 

D  x  Subjects  Within 

Groups 

1,881.000 

14 

134.357 

A  x  D 

1.687 

1 

1.687 

.0076 

T  x  A  x  D 

85.913 

1 

85.913 

.3874 

A  x  D  x  Subjects 

Within  Groups 

3,104.400 

14 

221.743 

Total 

12,271.937 

63 

*p  »  <  .05 
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Mean 


81.  86 
7  1. 74 

Difficulty  Levels 

"Easy"  75.57 

"Hard"  78.03 

One  partitioned  component  of  the  PERCOR  was  the  percentage  of 
the  radars  that  was  properly  turned  on  (PERON).  These  scores  were 
similarly  subjected  to  a  two-by-two-by-two  analysis  of  variance.  The 
variance  summary  is  shown  in  Table  4.  Only  the  aid  levels  variable  was 
statistically  significant.  The  probability  of  turning  on  the  correct  radar 
emitters  increased  in  the  aided  condition  and  decreased  in  the  manual 
condition.  This  seems  to  suggest  that  the  aid  facilitated  an  increase  in 
the  rate  of  turning  on  correct  emitters. 

There  was  also  a  tendency  toward  a  statistically  significant  inter¬ 
action  between  training  and  problem  difficulty  level.  This  tendency  is 
plotted  in  Figure  3.  The  PERON  score  did  not  tend  to  vary  across 
"easy"-"hard"  problems  for  the  trained  (experimental  group)  subjects 
but,  for  the  nontrained  (control  group)  subjects,  the  scores  tended  to  in¬ 
crease  markedly  between  "easy"  and  "hard.  "  Evidently,  the  training 
differentially  affected  the  tendency  to  activate  emitters  for  the  control 
as  compared  with  the  experimental  group.  Here,  we  note  that  informa¬ 
tion  denial  was  emphasized  during  the  training.  The  obtained  means 
were; 


Aiding  Levels 
Aid 

Manual 


Training  Levels  Mean 

Experimental  67.  17 

Control  66.  25 

Aiding  Levels 

Aid  72.73 

Manual  60. 69 

Difficulty  Levels 

"Easy"  64.67 

"Hard"  68.75 


The  final  partition  of  the  PERCOR,  the  PEROFF,  was  also  evaluated 
by  a  two-by-two-by-two  analysis  of  variance.  The  results  are  summarized 
in  Table  5.  The  only  statistically  significant  main  effect  was  aid  levels; 
neither  training  nor  difficulty,  nor  their  interactions  systematically  affected 
the  PEROFF.  The  means  were: 


EASY 


HARO 


Figure  3.  The  interaction  between  training  and  difficulty  in  the 
PERON  scores. 


Table  4 


Variance  Analysis  Summary  for  the  PERON  Scores 


Sum  of 

Source 

Squares 

Between  Groups 

4,221.484 

Training  (T) 
Subjects  Within 

31.629 

Groups 

4,189.845 

Within  Conditions 

20,912.250 

Aid  Use  (A) 

2,292.014 

T  x  A 

A  x  Subjects  Within 

107.643 

Groups 

5,578.093 

Difficulty  (D) 

260.014 

T  x  D 

D  x  Subjects  Within 

984.391 

Groups 

2,789.343 

A  x  D 

13.143 

T  x  A  x  D 

A  x  D  x  Subjects 

228.760 

Within  Groups 

8,656.845 

Total 

25,133.734 

Mean 


d.f. 

Squares 

F 

15 

1 

31.629 

0.11 

14 

299.275 

48 

1 

2,292.014 

5.75* 

1 

107.643 

0.27 

14 

398.435 

1 

260.014 

1.30 

1 

984.391 

4.94 

14 

199.239 

1 

13.143 

0.02 

1 

228.760 

0.37 

14 

618.346 

63 

*p-  <0.05 
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Table  5 


Variance  Analysis  Summary 

for  the 

PEROFF  Scores 

Sum  of 

Mean 

Source 

Squares 

d.f. 

Square 

F 

Between  Groups 

2,795.938 

15 

Training  (T) 
Subjects  Within 

10.568 

1 

10.568 

0.053 

Groups 

2,785.370 

14 

198.955 

Within  Conditions 

6,707.000 

48 

Aid  Use  (A) 

742.568 

1 

742.568 

4.960* 

T  x  A 

A  x  Subjects  Within 

52.550 

1 

52.550 

0.351 

Groups 

2,095.880 

14 

149.706 

Difficulty  (D) 

232.568 

1 

232.568 

1.866 

T  x  D 

D  x  Subjects  Within 

33.053 

1 

33.053 

0.265 

Groups 

1,745.380 

14 

124.670 

A  x  D 

52.553 

1 

52.553 

0.429 

T  x  A  x  D 

A  x  D  x  Subjects 

60.078 

1 

60.078 

0.491 

Within  Groups 

1,712.370 

14 

122.312 

Total 

9,502.938 

63 

*p  -  <0.05 
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Mean 


Training  Levels 

Experimental  90.58 

Control  91.52 

Aiding  Levels 

Aided  94.38 

Manual  87.72 

Difficulty  Levels 

"Easy"  89.25 

"Hard"  92.89 


The  analysis  of  the  relationship  of  activated  and  deactivated  emit¬ 
ters  between  the  plans  obtained  from  the  participants  and  those  of  the  cri¬ 
terion  plans  indicated  that  participants  more  frequently  errored  in  the  di¬ 
rection  of  leaving  radars  off  that  should  be  on.  They  might  be  described 
as  having  decision  rules  or  rule  governed  behavior  that  was  very  conser¬ 
vative.  Not  turning  on  radars  could  of  course  be  considered  an  extension 
of  denying  information  to  an  enemy.  Perhaps  the  participants  were  oper¬ 
ating  on  the  conjecture  that  the  best  way  to  avoid  giving  away  information 
about  task  force  composition  was  to  be  very  conservative. 

There  was  a  substantial  difference  between  a  conservative  plan  and 
■the  best  possible  plan.  The  aid  does  not  appear  to  address  this  difference. 
Perhaps  the  aid  should  allow  an  interactive  conversation  between  the  aid 
and  a  user  which  would  attempt  to  at  least  enlighten  an  individual  about  the 
possible  benefits  of  being  less  conservative. 

Summary  of  Variance  Analyses 

The  quality  of  the  EMCON  plans  developed  by  the  participants  was  eval¬ 
uated  by  studying  the  variance  of  a  number  of  measures.  The  FRAC  score 
was  used  to  evaluate  the  plans  among  themselves  and  the  DFRAC  and  PERCOR 
scores  (including  the  PERON  and  PEROFF  scores)  were  employed  to  evaluate 
the  plans  in  regard  to  a  criterion.  Consistently,  it  was  shown  that  use  or  non¬ 
use  of  the  EWAR  aid  in  solving  an  EMCON  problem  had  statistically  significant 
effects  on  the  scores.  In  FRAC  alone,  there  was  about  25  percent  increase 
when  the  aid  was  used.  Statistically  significant  differences  due  to  training  or 
to  problem  difficulty  level  were  not  evident.  These  latter  indications  may  be 
an  artifact  of  the  scoring  methods  employed- -especially  for  the  PERCOR, 
PERON,  and  PEROFF  scores.  There  was  almost  an  infinite  number  of  differ¬ 
ent  possible  EMCON  plans  that  potentially  could  be  developed  for  each  problem. 


Assessing  these  in  terms  of  a  set  of  binary  measures  is  similar  to 
scoring  a  musical  composition  against  a  set  of  binary  criteria.  It  seems 
that  development  of  a  symphony  was  asked  and  that  the  symphony  was 
graded  on  specific  criteria.  The  general  goal  allowed  for  a  great  deal 
of  artistic  variability  between  answers  and  the  variance  across  the  specif¬ 
ic  measures  (PERCOR,  PERON,  PEROFF)  may  simply  not  have  reflected 
this  artistic  latitude.  Such  a  situation  might  have  been  avoided  if  the  par¬ 
ticipants  were  asked  to  "select  the  best  EMCON  plan  from  the  alternate 
plans  A,  B,  C,  and  D"  as  opposed  to  "develop  a  best  EMCON  plan"  from 
the  almost  infinite  number  of  plans. 

Policy  Capturing 

During  the  data  acquisition,  each  participant  was  required  to  pro¬ 
duce  the  EMCON  plan  he  thought  was  best  for  eight  problem  situations. 

It  may  be  assumed  that  when  a  participant  made  a  decision,  it  was  based 
on  some  selected  set  of  values,  each  being  assigned  a  unique  importance 
or  weight.  Knowing  what  sets  of  EWAR  provided  data  were  emphasized 
by  the  participants  would  allow  insights  into  the  general  nature  of  EMCON 
decision  making  and  into  the  issues  of  the  relevance  of  the  data  supplied  by 
EWAR. 

In  order  to  ascertain  the  relationships  between  the  derived  plans 
and  the  associated  available  information,  a  set  of  multiple  regression 
analyses  was  completed.  In  these  analyses,  the  participant  plans  con¬ 
stituted  the  dependent  variable,  measured  in  terms  of  the  percentage  of 
correct  emitters  (PERCOR),  which  was  regressed  against  eight  independ¬ 
ent  variables.  Each  independent  variable  was  a  single  reference  number 
designed  either  to  summarize  the  data  about  a  plan  or  to  represent  an  index 
of  an  important  aspect  of  available  data. 

The  eight  independent  variables  were: 

surveillance  score  (SURV) 

probability  of  identifying  high  value  ship  (PID)* 
perceived  value  of  high  value  ship  (PERVAL)* 
information  denial  score  (INFO) 
trade  off  score  (TRADE) 

fraction  of  task  force  remaining  (FRACTF) 
fraction  of  high  value  ship  remaining  (FRACHVS) 
number  of  hits  across  both  threat  types  (HITS)* 

All  of  these  data  were  given  directly  or  indirectly  by  the  EWAR  aid. 
Two  threat  types  (SNEAKER  and  STREAKER)  were  assumed  and,  where  an 
asterisk  is  indicated  in  the  above  list,  the  reciprocal  of  the  actual  value 
was  employed  in  the  analysis. 
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The  multiple  regression  analyses  were  performed  on  the  overall 
data  collasped  across  conditions  and  on  the  data  fragmented  by  aid  levels, 
training  levels,  and  difficulty  levels. 

Overall  Analysis 


The  first  multiple  regression  analysis  was  based  on  the  data  col¬ 
lapsed  across  training,  aiding,  and  difficulty.  The  results  are  shown  in 
Table  6. 

Three  variables  entered  the  multiple  regression  equation- -the 
TRADE,  the  FRACTF,  and  FRACHVS.  Collectively,  they  accounted  for 
about  44  percent  of  the  total  variance. 

TRADE  was  the  fiz-st  variable  which  entered  the  equation  and  it 
yielded  an  R  of  .61,  accounting  for  37  percent  of  the  total  variance,  or 
about  84  percent  of  the  predictable  variance.  Each  of  the  other  variables 
as  they  were  entered  accounted  for  a  lesser  amount  of  the  variance.  With 
the  entry  of  FRACTF,  the  R  increased  to  .65,  accounting  for  an  additional 
five  percent  of  the  variance.  However,  the  entry  of  FRACHVS  had  only  a 
negligible  effect  on  total  predictability. 

It  appears  that  the  participants  based  their  decisions  primarily  on 
increasingly  high  TRADE  scores.  The  increasing  absolute  value  of  the 
TRADE  scores  was  evidence  of  a  more  finely  tuned  balance  between  the 
surveillance  coverage  and  information  denial.  After  arriving  at  a  finely 
balanced  plan,  the  results  suggest  that  minor  adjustments  may  have  been 
made  to  obtain  higher  FRACTF  scores.  The  adjustments  may  have  also 
resulted  in  some  minor  decreases  in  the  FRACHVS.  The  participants 
evidently  thought  that  the  most  important  aspect  of  EMCON  planning  was 
appropriately  balancing  the  divergent  needs  of  the  task  force  and  that  this 
should  be  done  in  such  a  way  as  to  produce  an  expectation  that  a  high  frac¬ 
tional  value  of  the  task  force  will  be  remaining. 

Aid  Levels 


The  next  multiple  regression  analyses  were  those  differentiated  on 
the  basis  of  the  aid  levels.  The  results  of  these  analyses  are  shown  in 
Table  7.  The  upper  half  of  Table  7  contains  the  data  from  the  aided  con¬ 
ditions  and  the  lower  half  the  data  from  the  unaided  conditions.  There 
were  virtually  no  differences  in  the  variables  entering  the  equation  and  in 
the  variance  accounted  for  between  the  aided  condition  and  the  overall  data. 
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Table  6 


Results  of  Overall  Multiple  Regression  Analysis 


Order  of  Variables 
Entering  Equation 

Multiple 

R 

R2 

R2 

Change 

Weight 

TRADE 

•  6103 

.3724 

.3724 

.3662 

FRACTF 

.6520 

.4251 

.0526 

.5242 

FRACHVS 

.6500 

.4350 

.0099 

.2289 

Table  7 

Results  of  the  Separate  Multiple  Regression 
Analyses  of  the  Aided  and  the  Unaided  Conditions 


Order  of  Variables 
Entering  Equation 

Multiple 

R 

trade 

.6020 

FRACTF 

.6358 

FRACHVS 

.6532 

FRACTF 

.5890 

TRADE 

.6181 

Aided 

5i 

R2 

Change 

B 

Weight 

.3625 

.3625 

.5283 

.4242 

.0418 

.6422 

.4262 

.0225 

.4153 

Unaided 

.3469 

.3469 

.3564 

.3820 

.0351 

.2630 

Constant 


-.0001 


Constant 


.0765 


-.1268 


54 


This  was  not  the  case  in  the  unaided  condition.  For  the  unaided 
condition,  the  FRACTF  variable  was  the  first  to  enter  the  equation.  Its 
entry  was  associated  with  an  R  of  .59,  accounting  for  about  35  percent  of 
the  variance.  The  only  other  variable  entering  the  equation  was  TRADE 
which  caused  the  R  to  increase  to  .  62 --a  four  percent  increase  in  pre¬ 
dictable  variance. 

These  data  suggest  that  EMCON  planning  was  based  on  a  different 
logic  when  the  participants  were  in  the  aided  and  in  the  unaided  conditions. 
EMCON  planning  was  very  strongly  influenced  by  the  FRACTF  scores  in 
the  unaided  condition,  and  not  by  the  TRADE  scores,  as  was  observed  in 
the  aided  condition.  Balancing  the  components  of  the  TRADE  scores  might 
have  required  some  very  fine  assessments  of  the  problems.  If  this  is  true, 
then  it  is  possible  that  the  aid  might  have  facilitated  these  assessments,  al¬ 
lowing  for  the  distribution  of  the  EMCON  plans  to  be  clearly  related  to  the 
distribution  of  the  TRADE  scores.  This,  of  course,  suggests  that  these 
types  of  fine  assessments  were  more  difficult  without  the  EWAR  aid.  What 
the  participants  seemed  to  have  done  in  the  manual  condition  was  to  select 
plans  intended  to  maintain  the  integrity  of  the  task  force  without  being  able 
to  develop  a  high  quality,  well  balanced  plan. 

Training 

The  next  regression  analyses  were  related  to  the  training  levels. 

The  results  of  these  are  presented  in  Table  8.  The  data  from  the  experi¬ 
mental  group  are  presented  at  the  top  of  the  table  and  the  control  group's 
data  are  presented  at  the  bottom.  Two  differences  are  apparent  from  a 
comparison  of  the  two  groups;  (1)  completely  different  variables  were 
entered  into  the  respective  regressions  equations,  and  (2)  substantial  dif¬ 
ferences  existed  in  the  amount  of  variance  accounted  for. 

Two  variables  entered  into  the  equation  as  a  result  of  the  regression 
analysis  of  the  experimental  group's  data.  The  first  was  FRACTF  which 
was  associated  with  an  R  of  .  49  and  accounted  for  24  percent  of  the  variance. 
When  PID  was  also  entered,  the  R  increased  to  .52  representing  a  three 
percent  increase  in  total  predictability.  The  total  variance  accounted  for, 

27  percent,  was  the  lowest  observed  in  any  of  the  current  regression  anal¬ 
yses. 

The  results  of  the  regression  analysis  of  the  control  group's  data 
were  very  different.  As  with  the  aided  and  overall  analyses,  the  TRADE 
variable  entered  first  and  accounted  for  62  percent  of  the  variance,  the 
largest  observed  in  any  of  the  current  regression  analyses.  The  total  var¬ 
iance  accounted  for,  with  the  HITS  and  PERVAL,  entered,  was  65  percent. 
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Table  8 

Results  of  the  Regression  Analyses  of  the 
Training  Levels  (Experimental  and  Control  Groups) 

Experimental 


Order  of  Variables 
Entering  Equation 

Multiple 

R 

R  2 

R2 

Change 

B 

Weight 

Constant 

FRACTF 

.4868 

.2369 

.2369 

.4787 

PID 

.5168 

.2672 

.0302 

.1731 

-.0050 

Control 

TRADE 

.7882 

.6212 

.6212 

.6806 

HITS 

.8023 

.6438 

.0225 

4.5508 

PERVAL 

.8062 

.6500 

.0063 

.1309 

-.9776 
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The  differences  between  groups  seem  to  suggest  that  the  training 
in  EMCON  planning  and  in  the  use  of  the  information  denial  guidelines 
affected  the  use  of  information.  The  experimental  group's  training  was 
intended  to  give  them  technologies  and  algorithms  which  would  enable  them 
to  generate  rather  close  approximations  of  the  information  that  was  pro¬ 
vided  by  the  EWAR  aid.  They  were  trained  in  estimating  the  surveillance 
coverage,  in  most  of  the  information  denial  aspects,  and  on  trade  offs. 
During  the  training,  the  participants  negatively  criticized,  on  the  basis  of 
incompleteness  and  lack  of  realism,  some  aspects  of  the  trade  off  algorithm 
contained  in  the  aid.  This  feeling  tone  may  have  negatively  biased  them  a- 
gainst  the  TRADE  score. 

Apparently,  because  the  control  group  was  not  exposed  to  the  train¬ 
ing,  its  members  were  not  biased  against  the  algorithm  and  the  resultant 
TRADE  score.  When  the  members  of  the  control  group  started  using  the 
EWAR  aid,  they  proved  to  be  surprisingly  sensitive  to  these  scores. 

Difficulty 

The  final  regression  analyses  were  concerned  with  the  difficulty 
levels.  The  results  of  these  analyses  are  presented  in  Table  9. 

The  total  variance  accounted  for  varied  between  difficulty  levels -- 
43  percent  for  the  "hard"  problems  and  52  percent  for  the  "easy"  problems. 
For  the  "easy”  problems,  the  initial  entry,  TRADE,  produced  a  R  of  .71 
and  accounted  for  50  percent  of  the  variance.  For  the  "hard"  problems,  the 
initial  entry,  FRACTF,  was  associated  with  a  R  of  .  64  and  represented  41 
percent  of  the  variance.  For  both  problem  difficulty  levels,  FRACTF  and 
TRADE  entered  the  regression  equations,  but  in  a  different  order.  It  ap¬ 
pears  that,  across  difficulty  levels,  the  quality  of  the  participants'  select¬ 
ed  plans  was  related  to  the  trade  off  analyses  but  the  TRADE  score  was  dif¬ 
ferentially  emphasized  as  a  function  of  problem  difficulty.  TRADE  scores 
were  more  heavily  emphasized  when  "easy"  problems  were  involved  and  the 
FRACTF  was  more  heavily  emphasized  in  the  thinking  of  the  participants 
when  "hard"  problems  were  involved. 

Discussion  of  Regression  Results 


Consistently,  the  regression  results  indicated  that  the  participants 
employed  "second  order"  information  in  deriving  EMCON  plans.  By  "sec¬ 
ond  order"  information  we  mean  derived  or  aid  generated  information  as 
opposed  to  the  basic  surveillance  and  information  denial  information  which 
are  a  direct  function  of  emitters  activated  or  deactivated.  While  different 
sets  of  information  may  have  been  more  or  less  emphasized  across  con¬ 
ditions  (as  was  to  be  anticipated),  in  no  case  did  the  participants  revert  to 
raw  emitter  status  as  a  decision  basis.  Such  a  finding  supports  the  need  for 
and  utility  of  some  type  of  decision  aiding  in  the  EMCON  context  and  of  the 
EWAR  aid. 


Table  9 

Results  of  the  Regression  Analyses  of  Difficulty  Levels 


"Easy" 


Order  of  Variables 
Entering  Equation 

Multiple 

R 

R2 

Change 

B 

Weight 

Constant 

TRADE 

.7080 

.5012 

.5012 

.6767 

FRACHVS 

.7154 

.5118 

.0106 

.2878 

FRACTF 

.  7226 

.5222 

.0104 

.2352 

.0156 

"Hard" 

FRACTF 

.6439 

.4146 

.4146 

.5422 

V.IADE 

.6526 

.4259 

.0113 

.1444 

-.0085 
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Linear  Structural  Equation  Model 


The  regression  analyses  suggested  that  the  variables  most  clearly 
associated  with  EMCON  plan  quality  were  "second  order"  functions  such 
as  TRADE  scores. 

The  suggestion,  however,  says  no  more  than  that  a  trade  off  was 
made.  While  pragmatic,  it  does  not  yield  an  appreciation  of  what  psy¬ 
chological  acceptances  or  rejections  were  involved.  To  develop  fuller  in¬ 
sight  into  the  nature  of  the  trade  offs,  a  structural  equation  trade  off  or 
"causal"  model  was  developed.  This  model  utilized  as  basic  a  set  of  as¬ 
sumptions  as  possible,  while  incorporating  what  appeared  to  be  the  major 
determinants  of  an  EMCON  trade  off.  The  model,  which  is  described  in 
detail  in  Appendix  C,  considered  five  variables;  trade  score,  surveillance 
score,  information  denial  score,  need  for  protection,  and  need  for  security. 

The  data  from  the  aided  conditions  were  employed  to  calculate  struc¬ 
tural  coefficients  by  the  least  squares  technique.  The  Statistical  Package 
for  the  Social  Sciences  (SPSS)  program  (Nie,  Hall,  et  al.  ,  1975)  was  employ¬ 
ed  to  this  end.  Appendix  C  contains  calculational  details. 

The  path  coefficients,  so  derived,  may  be  interpreted  in  terms  of 
causality,  i.  e.  ,  a  change  of  one  unit  in  variable  x_  causes  a  change  in 
units  in  variable  z.  The  coefficients  for  surveillance  score  and  information 
denial  score  on  trade  score  were  respectively  .79  and  .25.  This  suggests 
positive  causal  relationships  for  the  two  variables  (surveillance  and  infor¬ 
mation  denial)  with  the  major  consideration  being  surveillance  coverage. 

There  was  also  a  low  reciprocal  relationship  (-20)  between  surveil¬ 
lance  and  information  denial.  This  result  represents  a  cognitive  processing 
of  the  maximizing  tendencies  set  into  effect  by  the  need  for  protection  and 
the  need  for  security. 

Discussion  of  Structural  Model 


The  findings  suggest  that  the  composition  of  the  EWAR  aid  is  such  as 
to  support  the  psychological  (latent)  and  logical  needs  for  developing  an  effec¬ 
tive  EMCON  plan.  It  is  not  known  whether  or  not  such  a  model  was  implicit 
in  the  thinking  of  the  aid's  developers.  Nevertheless,  it  seems  that  the  aid’s 
composition  is  such  as  to  support  the  processes  employed  by  our  sample  to 
develop  their  plans.  To  this  extent,  the  EWAR  aid  may  be  held  to  provide 
direct  support  to  the  cognitive  and  psychological  needs  in  EMCON  planning. 

It  goes  without  saying  that  a  causative  model  should  be  implicit  in  the  think¬ 
ing  of  the  developer  of  any  decision  aid.  Moreover,  it  is  quite  possible  that 
such  a  model  should  be  made  explicit  before  any  decision  aid  is  developed. 
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A  preconceived  model  would  tend  to  minimize  the  proliferation  of  aids  and 
aid  features  which  contain  a  wide  variety  of  information  of  marginal  value 
to  the  user.  In  such  a  case,  the  test  of  the  aid  might  be  based  on  the  ex¬ 
tent  of  causality  indicated  by  the  model. 


Confidence,  Difficulty,  and  Learning 

When  he  completed  each  problem,  each  participant  was  required  to 
make  a  judgment  about  the  difficulty  of  the  problem  and  his  confidence  in  his 
decision.  The  relationship  between  these  two  ratings  was  examined.  The 
relationship,  as  indicated  by  the  Pearson  product-moment  correlation,  was 
slightly  negative  (r  »  -.  18).  Confidence  seems  to  have  been  only  slightly  af¬ 
fected  by  the  perceived  problem  difficulty.  The  relationship  varied  with  train¬ 
ing.  The  correlation  was  more  negative  for  the  experimental  group  (r=  -.  33) 
than  for  the  control  group  (r  =  -.16). 

The  relationship  between  the  perceived  and  a  priori  difficulty  ratings 
was  also  evaluated.  The  results  indicated  virtually  no  relationship  (r  =  .  15). 

In  addition,  a  breakdown  by  groups  revealed  only  minor  influences;  r  =  .  12 
and  .  18  for  the  experimental  and  the  control  groups,  respectively. 

Learning 

The  FRAC  data  were  analyzed  for  a  learning  effect  by  a  linear  regres¬ 
sion  analysis.  For  each  participant,  the  FRAC  scores  were  organized  in  the 
serial  order  in  which  the  associated  problems  were  presented  during  the  test¬ 
ing.  These  data  were  analyzed  for  any  systematic  variablity  across  the  ordinal 
positions.  The  analysis  indicated  that  a  negligible,  non-significant  effect  was 
apparent,  r=-.05.  That  is,  there  was  no  apparent  learning  effect. 


Multiattribute  Utility  Analyses 

The  multiattribute  utility  analytic  technique  is  devised  to  allow  quan¬ 
tification  of  the  utility  of  a  process  or  product  when  multiple  goals  and  mul¬ 
tiple  attributes  are  involved,  as  in  the  case  of  the  EWAR  aid. 

The  participants  rated  the  seven  EWAR  aid  goals,  shown  in  Exhibit  18, 
on  how  closely  each  was  attained  by  the  aid.  To  this  end,  they  distributed  100 
points  across  the  goals.  These  ratings  were  used  in  combination  with  a  set 
of  weights,  selected  to  reflect  the  importance  of  each  goal,  to  obtain  estimates 
of  how  well  the  EWAR  aid  achieved  its  purposes  or  goals  in  relt  tion  to  their 
importance. 
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The  utilities  were  obtained  by  the  formula: 


n 

GUi  =  2 

i  =  1 

where  GUj  indicates  the  perceived  utility  of  goal  i,  and  Aj  and  w^  are  the 
participants'  achievement  ratings  and  the  developer's  weights  of  the  im¬ 
portance  of  goal  i,  respectively.  The  utilities  are  presented  in  Table  10. 

In  Table  10,  the  first  four  rows  of  data  under  each  goal  contain  the  optimum 
utility,  the  overall  utility,  the  utility  from  the  experimental  group,  and  the 
utility  from  the  control  group,  respectively.  The  next  three  and  the  last 
three  rows,  respectively,  contain  the  percentage  of  each  goal's  attainment 
and  the  mean  ratings  based  on  the  overall  data,  and  the  experimental  and 
the  control  groups'  individual  data. 

The  optimum  utilities  represent  criteria  against  which  the  perceived 
utilities  may  be  judged.  The  optimum  utility  measures  rest  on  the  assump¬ 
tion  that  goals  should  be  achieved  in  proportion  to  their  importance.  Over- 
achieving  a  goal  represents  an  uneconomical  allocation  of  resources  and 
underachievement  represents  a  deficiency.  If  each  goal  was  perceived  to  be 
achieved  at  an  optimum  level,  then  the  ratings  of  achievement  would  be  equal 
to  the  weights  of  importance.  Therefore,  the  optimum  was  produced  using 
the  goal  utility  formula  but  with  A^  equal  to  Wi. 

The  first  utility  to  be  examined  was  based  on  the  combined  data  of 
the  experimental  and  the  control  groups  (overall  row  in  Table  10).  Goal  Gi 
was  one  of  the  less  important  goals  of  the  EWAR  aid.  It  possessed  a  weight 
of  10  and  had  an  overall  perceived  utility  of  810.  A  comparison  of  this  value 
with  the  optimum  utility  of  800  indicates  that  the  aid  achieved  its  goal  of  fa¬ 
cilitating  the  formation  of  hypotheses  about  how  best  to  improve  a  plan  at 
about  the  optimum  level.  Goal  G2  was  the  most  important  goal  of  the  EWAR 
aid.  Goal  G2  possessed  a  weight  of  20.  The  overall  perceived  utility  was 
2640  which  represents  an  underachievement  of  17  percent  when  compared 
with  the  optimum  of  3200.  An  83  percent  goal  attainment  represents  reason¬ 
able,  but  not  full,  success  relative  to  the  aid's  most  important  goal--pro- 
ducing  better  EMCON  plans. 

The  third  goad,  G3,  was  moderately  important.  It  possessed  a  weight 
of  15  and  obtained  an  overall  utility  rating  of  1275.  Comparing  this  to  the 
optimum,  1800,  suggests  that  this  goal  was  less  well  realized  than  the  two 
previously  described  goals.  The  goal  of  helping  to  quantify  the  surveillance 
and  information  consequences  of  a  plan  had  a  71  percent  attainment.  This 
was  the  lowest  attainment  percentage  ol  any  of  the  goals. 

Goal  G^,  to  make  it  practical  to  examine  a  larger  set  of  alternative 
plans,  also  possessed  a  lesser  importance  weight  (10).  The  perceived 
utility,  1470,  was  184  percent  of  the  optimum  utility  of  800.  This  suggests 
considerable  overachievement.  Certainly,  the  hard  copy  features  of  the 
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EWAR  aid  facilitated  the  consolidation  of  data  which  could  be  used  for  com¬ 
parative  purposes.  In  addition,  the  TRADE,  SCORE  display  may  have  con¬ 
tributed  to  the  very  high  utility  of  G4.  The  TRADE,  SCORE  was  specifically 
designed  to  assist  in  making  comparisons  across  plans.  Presumably,  it 
allowed  the  choice  of  finely  balanced  plans. 

The  fifth  goal,  G5  (to  understand  how  different  factors  contribute  to 
EMCON  planning),  was  moderately  important.  It  possessed  a  weight  of  15. 
The  perceived  utility  of  the  EWAR  relative  to  this  goal  was  1305.  This 
value  is  somewhat  lower  than  the  optimum  value  of  1800.  The  73  percent 
attainment  level  was  about  as  low  as  that  observed  for  G3.  That  the  goal 
of  understanding  how  different  factors  (threats,  radars,  etc.)  contribute 
to  EMCON  planning  was  only  moderately  achieved  may  signify  a  need  for  a 
more  direct  means  of  assessing  the  effects  of  changing  physical  factors. 

This  may  be  relevant  to  the  low  relative  rating  of  G3.  A  prior  report 
(Siegel  &  Madden,  1980)  suggested  that  feedback  functions  within  an  aid  act 
to  sensitize  users  to  relevant  physical  factors  in  the  situation  under  con¬ 
sideration.  The  EWAR  aid  seems  to  lack  any  simple,  direct  feedback 
mechanism  relative  to  physical  features. 

The  sixth  goal,  Gg,  of  the  EWAR  aid,  was  to  improve  confidence  in 
the  quality  of  the  user's  EMCON  plans.  This  goal  received  an  overall  utility 
of  1680  against  an  optimum  utility  of  1800.  Accordingly,  this  goal  had  a  93 
percent  attainment  level  which  compares  favorably  with  the  level  attained 
by  Gp  Gi  was  the  only  other  goal  for  which  the  attainment  approximated 
the  optimum. 

The  final  goal,  G7,  was  also  of  moderate  importance  and  was  over¬ 
achieved.  Its  overall  utility  was  2340.  This  value  represents  an  attainment 
of  130  percent  of  the  optimum.  The  goal,  to  facilitate  the  comparison  of  al¬ 
ternative  plans,  to  some  extent  was  not  mutually  exclusive  from  G4.  Goal 
G7's  overachievement  may  be  related  to  the  same  factors  described  in  re¬ 
lation  to  overachievement  of  G4. 

Comparing  the  utilities  across  groups  indicated  some  variability 
although  the  individual  group  distributions  across  goals  tended  to  be  sim¬ 
ilar  to  the  overall  distribution.  Generally,  the  experimental  (trained) 
group  rated  four  of  the  goals  (Gj,  G3,  G5,  and  Gg)  higher  than  the  control 
(untrained)  group.  The  largest  difference  was  observed  for  goal  Gj.  The 
experimental  group  rated  Gj  (to  facilitate  hypotheses  about  how  best  to  im¬ 
prove  a  current  plan)  as  being  overachieved  by  25  percent,  while  the  control 
rated  Gj  as  being  22  percent  underachieved.  This  result  may  be  due  to  the 
training  which  stressed  formulating  hypotheses  about  improving  plans  in 
two  ways:  (1)  in  the  general  discussion  of  manual  planning,  and  (2)  in  the 
description  of  the  information  denial  rules. 
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Other  differences  in  the  perceived  utility  between  groups  related  to 
the  overattained  goals,  G4  and  G7.  In  both  cases,  the  experimental  group's 
ratings,  although  above  the  optimum,  were  considerably  below  the  control 
group's.  This  result  may  also  be  related  to  training  and  be  an  expression 
of  the  fact  that  the  experimental  group  may  have  appreciated  more  subtle 
aspects  of  the  aid  as  compared  with  the  more  obvious  comparison  facilita¬ 
tions. 

Summary  of  the  Utility  Analysis 

The  goals  of  the  EWAR  decision  aid  were  reasonably  well  attained. 

The  aid  apparently  lent  itself  rather  well  to  supporting  hypothesis  generation 
and  to  confidence  assurance.  It  was  also  perceived  to  be  somewhat  success¬ 
ful  relative  to  the  goal  of  producing  better  EMCON  plans.  However,  there 
was  apparently  some  inability  on  the  part  of  the  aid  to  make  the  effects  of 
physical  factors  evident  and  surveillance  and  information  consequences  clear. 
The  aid  overachieved  the  goals  which  were  concerned  with  the  comparison 
of  plans.  Further  emphasis  on  features  relating  to  the  plan  comparison 
goals  during  future  development  of  the  aid,  does  not  seem  warranted. 

We  note  that  some  features  of  the  EWAR  aid  were  not  included  in  the 
current  test.  It  is  not  believed  that  the  inclusion  of  these  features  would 
affect  the  utility  scores  to  any  marked  degree.  However,  the  possibility  of 
such  an  effect  can  not  be  ruled  out. 

Postevaluation  Interview 


At  the  conclusion  of  the  data  acquisition,  the  participants  were  in¬ 
terviewed  about  their  personal  reactions  to  the  EWAR  aid,  its  value,  and 
how  it  might  be  improved.  The  interview  was  semiformal  in  nature.  The 
full  details  of  the  interview  content  are  presented  as  Appendix  A  to  this 
report. 

Usefulness  Ratings 

At  the  opening  of  the  postevaluation  interview,  each  participant  was 
asked  to  rate  the  usefulness  of  each  of  the  various  EWAR  displays.  Each 
display  was  rated  by  way  of  a  five  category  rating  scale  which  was  calibrated 
as;  (1)  minimally  useful,  (2)  somewhat  useful,  (3)  moderately  useful,  (4) 
considerably  useful,  (5)  extremely  useful. 

The  mean  usefulness  ratings  for  each  display  are  shown  in  Table  11. 
In  Table  11,  the  displays  are  organized  by  type  across  the  top  of  the  table. 

In  addition,  the  data  are  collapsed  across  the  groups  in  the  top  row  (over¬ 
all)  and  they  are  differentiated  on  the  basis  of  training  in  the  second  and 
third  rows. 
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Usefulness  Ratings  of  the  EWAR  Displays 
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Collapsed  across  groups,  the  highest  ratings  were  obtained  for  two 
of  the  trade  off  displays  and  the  EMCON  display.  The  TRADE,  STRIKE,  the 
TRADE,  SCORE,  and  the  DIS  EMCON  displays  were  assigned  mean  useful¬ 
ness  ratings  of  4.  82,  4.07,  and  4.  06  (considerably  useful),  respectively. 
That  the  TRADE,  STRIKE  and  the  TRADE,  SCORES  were  rated  highest  is 
concordant  with  the  results  of  the  policy  capturing  analysis.  As  previously 
reported,  the  TRADE,  SCORE  along  with  two  components  of  the  TRADE, 
STRIKE--the  FRACTF  and  the  FRACHVS--rather  consistently  entered  the 
multiple  regression  equations  and  in  various  combinations  accounted  for 
most  of  the  predictable  variance.  The  congruence  between  the  results  of 
the  policy  capturing  and  usefulness  ratings  reinforces  the  contentions  con¬ 
cerning  the  emphasis  on  these  displays  by  the  participants. 

The  high  ratings  observed  for  the  EMCON  plan  display  seems  to  re¬ 
flect  its  singular  nature.  The  display  provided  no  additional  information 
but  it  organized  the  important  ship  by  radar  type  data  in  a  very  logical  way. 

It  may  be  that  this  served  an  important  function,  helping  to  generate  hypo¬ 
theses  as  to  what  radars  should  be  turned  on  and  off.  The  utility  of  displays 
which  serve  an  information  organizational  purpose  within  a  decision  aid  was 
previously  recognized  (Siegel  &  Madden,  1980)  and  the  present  finding  rein¬ 
forces  this  prior  conclusion. 

The  lowest  rating,  1.88  (minimally  useful),  was  observed  for  the 
INFO,  DETAILS  display,  while  the  SURV,  MAX  and  TRADE,  MAP  were 
rated  only  slightly  higher  with  ratings  of  2.06  and  2.56  (somewhat  useful), 
respectively. 

Considering  only  the  surveillance  coverage  and  information  denial 
groupings,  within  each  group  each  of  the  displays  may  be  conceived  as  pre¬ 
senting  increasingly  abstract  or  "second  order"  assessments  along  the 
relevant  dimensions.  The  least  abstract  of  these  were  the  SURV,  MAX 
and  INFO,  DETAILS  along  the  surveillance  and  information  denial  dimen¬ 
sions,  respectively.  From  the  relatively  low  ratings  given  these  two  dis¬ 
plays,  one  may  infer  that  the  participants  did  not  find  the  relatively  funda¬ 
mental  displays  to  possess  high  value.  However,  the  usefulness  of  the 
surveillance  coverage  and  information  denial  displays  were  nonmonotonic 
functions  of  the  order  or  level  of  abstraction  of  the  data  presented  in  the 
display.  The  most  useful  were  the  intermediately  complex  SURV,  CUM 
and  INFO,  TARGETS  displays.  Of  intermediate  usefulness  were  the  most 
abstract,  the  SURV  and  INFO  scores. 

A  relatively  low  rating  was  given  to  the  TRADE,  MAP  display,  a 
display  which  can  not  be  considered  to  represent  a  low  level  of  abstraction. 
However,  the  low  rating  of  the  TRADE,  MAP  display  probably  was  a  special 
case.  According  to  the  statements  made  by  several  of  the  participants,  this 
display  was  not  used  because  it  was  redundant,  i.  e. ,  it  represented  a  re¬ 
statement  of  the  data  from  the  SURV,  CUM  and  INFO,  TARGET  displays. 
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Generally,  there  was  a  tendency  for  the  experimental  group  to  rate 
the  DIS,  EMCON  and  the  trade  off  displays  somewhat  higher  than  the  con¬ 
trol  group.  The  reverse  was  true  for  the  surveillance  and  information 
displays  where  the  control  group  tended  to  assign  ratings  which  were  higher 
than  those  of  the  experimental  group.  However,  comparing  the  distributions 
of  the  ratings  across  displays  indicates  considerable  similarity  both  between 
groups  and  with  the  distribution  of  the  overall  ratings. 

Generally,  the  preference  was  for  displays  at  some  level  of  com¬ 
plexity;  i.  e. ,  displays  which  presented  "second  order"  or  processed  infor¬ 
mation. 

Display  Groups 

In  addition  to  the  ratings  of  the  individual  displays,  the  participants 
rated  the  usefulness  of  each  of  the  four  general  display  groups.  These 
ratings,  shown  in  the  lower  half  of  Table  11,  largely  concur  with  the  indi¬ 
vidual  display  ratings.  The  trade  off  group  (a  "second  order"  group)  was 
rated  highest.  This  preference  was  followed  by  the  display  EMCON  group, 
while  the  information  and  surveillance  groups  were  about  tied  for  the  low¬ 
est  positions.  The  trained  subjects  rated  the  first  two  (trade  off  and 
EMCON  groups)  higher  than  the  untrained  and  the  reverse  held  true  for  the 
latter  two  (information  and  surveillance  groups).  Since  the  displays  in  the 
information  and  the  surveillance  groups  were  based  on  lower  order  infor¬ 
mation  than  the  displays  in  the  trade  off  group,  it  may  be  that  the  training 
sensitized  the  participants  to  the  utility  of  the  higher  order  information. 

Rank  Ordering 

The  participants  were  also  asked  to  rank  order  the  displays  within 
each  group  and  the  groups  in  general.  These  rankings  were  obtained  be¬ 
cause  it  was  thought  that,  if  the  ratings  were  inconclusive,  then  at  least 
the  rankings  would  show  gross  ordinal  measures  of  usefulness.  As  indi¬ 
cated  above,  the  rating  data  were  far  from  inconclusive.  Accordingly,  the 
rankings  are  somewhat  redundant. 

The  results  of  the  ranking,  shown  in  Table  12,  indicate  considerable 
consistency  with  the  prior  data.  At  the  top  of  the  table,  the  rank  order  of 
the  surveillance  displays  is  shown  for  the  overall  data  and  differentiated  on 
the  basis  of  training.  Very  consistently,  the  CUM  display  was  ranked  num¬ 
ber  1  with  SCORE  display  ranked  number  2  and  MAX  ranked  3.  The  same 
consistency  with  the  prior  data  was  observed  in  the  information  displays. 

The  TARGET,  the  SCORE,  and  the  DETAILS  were  ranked  1,  2,  and  3  re¬ 
spectively.  Again  with  the  trade  off  analyses,  the  STRIKE,  the  SCORE, 
and  the  MAP  displays  were  consistently  ranked  1,  2,  and  3. 
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Table  12 


Mean  Rank  Order  of  the  EWAR  Displays  According  to  their  Usefulness 

Surveillance 

MAX 

Overall 

3 

Experimental 

3 

Control 

3 

CUM 

1 

1 

1 

SCORE 

2 

2 

2 

Information 

DETAILS 

3 

3 

3 

TARGET 

1 

1 

1 

SCORE 

2 

2 

2 

Trade  Off 

MAP 

3 

3 

3 

SCORE 

2 

2 

2 

STRIKE 

1 

1 

1 

Mean  Rank  Order  of  the  Four  Display  Categories 


Display,  EMCON 

Overall 

2 

Experimental 

2 

Control 

A 

Surveillance 

3 

3 

3 

Information 

A 

A 

2 

Trade  Off 

1 

1 

1 

The  ranking  of  the  displays  by  category  showed  less  agreement 
with  the  prior  data  based  on  ratings.  Although  the  trade  off  display  group 
was  again  always  ranked  number  1,  the  ranking  obtained  from  the  pooled 
data  assigned  rank  2  to  the  EMCON  display,  rank  3  to  the  surveillance 
group,  and  rank  4  to  the  information  group.  This  same  ordering  was  also 
observed  for  the  experimental  data.  However,  the  control  group's  data 
were  different.  The  number  2  rank  was  assigned  to  the  information  group; 
the  3  rank  was  assigned  to  the  surveillance  group,  and  the  lowest  rank  was 
assigned  to  EMCON. 

Other  Opinions 

Each  participant  was  asked  a  number  of  open  ended  questions  in  the 
course  of  the  interview.  The  open  ended  questions  were  grouped  into  four 
general  areas;  (1)  the  usefulness  and  advantages/disadvantages  of  the  aid, 
(2)  the  sufficiency/ insufficiency  of  the  information  provided,  (3)  personal 
strategies  and  heuristics  used  by  the  participants  in  producing  an  EMCON 
solution,  and  (4)  the  assumptions  on  which  the  EWAR  aid  rested. 

The  participants  generally  found  the  aid  to  be  very  useful  and  about 
60  percent  indicated  that  the  aid  was  more  useful  for  some  problems  than 
for  others.  The  reasons  suggest  that  "the  simple  problems  did  not  re¬ 
quire  computer  treatment,"  and  that  "the  aid  was  slower  for  some  prob¬ 
lems.  "  Implied  by  the  "simple"  problems  were  the  easy  problems  with 
"few  ships  and  few  radars.  "  This  opinion,  however,  was  not  universal  and 
some  participants  thought  the  aid  to  be  equally  useful  across  all  of  the  prob¬ 
lems. 


When  asked  to  compare  the  plans  developed  with  the  aid  against  those 
developed  manually,  plans  developed  with  the  help  of  the  EWAR  aid  were 
said  to  be  superior.  About  67  percent  of  the  responses  indicated  that  the 
use  of  the  aid  resulted  in  better  EMCON  plans.  Most  of  the  participants 
said  that  using  the  aid  allowed  for  more  "finesse"  in  planning,  producing 
"better,  "  more  "sophisticated"  plans  and  with  an  increased  "confidence.  " 
Those  participants  who  thought  that  there  was  little  difference  between  plans 
developed  with  or  without  the  use  of  the  EWAR  aid  indicated  that  they  had 
learned  so  much  using  the  aid  that  they  could  now  produce  plans  of  equal 
quality  without  it.  These  participants  stated  that  "after  getting  a  feel  for 
the  trade  offs  between  information  denial  and  surveillance,  the  marginal 
value  of  using  the  aid  was  small"  and  that  "after  working  with  the  aid  you 
get  basic  insights  and  with  minor  corrections  you  get  the  same  results  with 
or  without  the  aid.  "  Furthermore,  they  said  that  they  "took  advantage  of 
what  was  learned  with  the  aid  to  apply  strategies.  "  Such  findings  support 
the  utility  of  the  aid  as  a  training  device. 

When  asked  if  one  technique  was  better  than  the  other,  77  percent 
affirmed  that  the  aid  represented  a  better  technique.  The  reasons  given 
were  the  possibility  of  "comparing  numerous  things  simultaneously,  "  "in¬ 
creasing  flexibility"  and  the  "number  of  plans  that  could  be  compared.  " 
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Those  who  denied  the  superiority  of  EMCON  plans  produced  with  the  assist¬ 
ance  of  the  EWAR  aid  based  their  arguments  on  the  large  amount  of  transfer 
from  aided  to  nonaided  development  and  suggested  that  under  those  con¬ 
ditions  "neither  technique  was  better.  " 

Most  of  the  participants  said  that  the  differences  between  techniques 
were  quantitative  rather  than  qualitative  in  nature.  That  is,  the  aid  "does 
more  at  once,  "  giving  a  "more  exact  idea  of  probabilities  and  exact  coverage.  " 
The  major  differences  centered  on  the  "ability  to  do  more  quantitative  analyses.  " 
Put  another  way,  "gross  assumptions  were  made  manually"  which  were  "not 
constrained"  as  they  were  when  using  the  aid.  Other  important  differences  had 
to  do  with  the  graphics --"ability  to  get  a  visual  representation  of  the  plan.  " 

All  of  the  participants  said  that  the  aid  increased  their  confidence  in 
the  plans  they  developed  and  all  also  trusted  information  claculated  by  the 
aid  more  than  that  which  they  calculated  themselves. 

Other  questions  concerned  the  use  of  the  EWAR  aid  under  live  and 
real  combat  conditions.  Ninety  four  percent  of  the  participants  answered 
that  they  would  feel  comfortable  using  the  aid  in  such  conditions.  This 
primarily  meant  that  the  aid's  information  would  represent  another  set  of 
information  that  must  be  considered  by  an  EW  officer.  Some  also  indicated 
that  the  aid  would  facilitate  planning  because  it  was  "better  than  nothing,  " 
that  they  would  "feel  better  with  number  s,  "  and  the  aid  would  probably  "help 
make  a  better  decision.  "  On  the  other  hand,  some  respondents  thought  that 
the  aid  needed  "major  modifications"  because  it  was  "much  too  slow"  and 
"tried  to  do  too  much.  " 

Sufficiency  of  Information  Provided 

The  second  major  area  about  which  detailed  information  was  devel¬ 
oped  during  the  interview  concerned  the  adequacy  of  the  displays,  the  suf¬ 
ficiency  of  the  data  made  available,  and  the  mechanics  of  the  command 
structure.  About  69  percent  of  the  participants  thought  that  one  or  more 
of  the  graphic  displays  was  confusing.  One  display  that  generally  was 
thought  to  be  confusing  was  the  SURV,  CIRCLES,  especially  when  the  threat 
was  Streaker. 

The  participants  also  provided  information  about  features  that  were 
thought  to  be  needed.  One  change  which  was  suggested  several  times  em¬ 
phasized  that  the  aid  "should  provide  optimum  plans"  or  that  the  "computer 
make  reasonable  suggestions  as  to  what  to  try.  "  It  was  generally  thought  that 
a  lot  of  time  was  wasted  getting  into  a  position  from  which  good  plans  could 
be  developed  and  that  the  aid  should  supply  "some  starting  points.  " 
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The  other  two  items  most  criticized  were  mechanical  features  of  the 
aid.  One  concerned  the  necessity  of  saving  scores  for  future  use  by  the 
trade  off  analysis.  The  participants  were  somewhat  vexed  by  the  time  con¬ 
sumed  by  the  requirements  to  save  scores  and  indicated  that  "every  time 
the  plan  is  changed,  the  scores  should  be  automatically  saved.  "  The  other 
criticism  in  this  area  was  concerned  with  the  difficulty  of  the  command 
structure  and  the  "inadequacy  of  the  error  messages.  "  A  complete  "re¬ 
structuring"  of  the  command  and  rethinking  of  the  error  messages  were 
generally  thought  to  be  needed. 

Other  improvements  which  were  less  consistently  suggested  included: 
(1)  a  comparison  table  for  the  TRADE,  STRIKE,  (2)  concurrent  presentation 
of  INFO  and  SURV  SCORES,  (3)  presentation  of  threat  vectors  on  the  SURV, 

CIR  and  CUM  displays  and  TRADE,  MAP  displays,  (4)  combination  of  INFO, 
DETAILS  and  TARGETS  so  that  there  would  be  real  numbers  on  the  latter, 

(5)  presentation  of  the  differential  effects  of  adding  one  or  more  radars  (a 
sensitivity  function).  Each  of  these,  as  for  the  modification  of  the  structure 
of  the  command  sequence/language,  .vould  add  user  convenience.  Such  con¬ 
venience  features  seem  to  be  important  to  the  users  of  decision  aids. 

The  participants  were  also  asked  if  any  of  the  EWAR  aid's  displays 
were  not  necessary.  Two  displays  were  mentioned  about  50  percent  of  the 
time,  the  INFO,  DETAILS  and  TRADE,  MAP.  The  former  was  mentioned 
because  the  information  it  presented  was  adequately  handled  at  a  higher  level 
by  the  INFO,  TARGET  display  and  the  latter  because  it  was  redundant,  had 
"no  function,  "  and  was  "difficult  to  read.  "  If  one  remembers  that  these  dis¬ 
plays  were  consistently  given  the  lowest  usefulness  ratings,  then  their  men¬ 
tion  here  should  not  be  surprising. 

Strategies  and  Heuristics 

The  thii'd  general  area  of  inquiry  concerned  the  strategies  and 
heuristics  used  by  the  participants. 

The  first  topic  in  this  area  was  concerned  with  the  information  denial 
guidelines.  These  were  provided  only  to  the  experimental  group.  The  par¬ 
ticipants  from  this  group  were  asked  how  useful  the  rules  were  in  developing 
and  evaluating  EMCON  plans  and  how  much  influence  they  had  on  plan  devel¬ 
opment.  To  both  of  these  questions,  the  participants  most  frequently  answered 
"moderately  useful  and  influential.  "  In  addition,  they  indicated  that  the  rules 
made  only  a  moderate  contribution  to  their  overall  EMCON  planning.  The 
reasons  given  were  related  to  the  "common  sense"  nature  of  the  rules;  most 
indicated  that  the  "concept  is  good  for  teaching  purposes,  but  it  is  intuitive.  " 

When  all  the  participants  from  both  groups  were  asked  what  strategies 
or  heuristics  they  used  to  derive  their  best  plan,  the  answers  were  the  same. 
Regardless  of  the  group,  the  participants  overwhelmingly  started  a  problem 
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by  hiding  the  high  value  ship.  They  did  this  by  essentially  applying  the 
information  denial  rules.  That  is,  even  the  control  group  participants, 
listed  steps  which  to  all  intents  and  purposes  were  statements  of  the  rules. 
The  responses  indicated  that  once  the  insight  was  developed  that  the  radars 
on  the  high  value  ship  should  be  deactivated,  the  rest  of  the  problem  be¬ 
came  a  game  in  which  the  goal  was  to  maximize  the  fractional  value  remain¬ 
ing  along  with  the  surveillance  and  information  scores. 

The  participants  were  also  questioned  about  any  differences  in  their 
approaches  to  EMCON  plan  development  as  opposed  to  plan  evaluation.  The 
displays  most  often  used  for  plan  development  were  the  SUKV,  CUM  display 
(indicated  by  about  40  percent  of  the  participants  to  be  important)  and  the 
TRADE,  SCORES  (indicated  by  30  percent  as  important).  Also  mentioned 
about  10  percent  of  the  time,  were  the  EMCON  plan  display,  the  INFO, 
TARGET,  and  the  SURV,  CIRCLES.  One  display,  the  TRADE,  STRIKE, 
predominated  in  plan  evaluation. 

These  data  suggest  that  the  importance  of  the  TRADE,  SCORE  and 
TRADE,  STRIKE  were  derived  from  different  sources.  The  TRADE, 

SCORE  apparently  was  important  because  it  helped  generate  hypotheses 
and  the  TRADE,  STRIKE  was  important  because  it  helped  test  the  hypoth¬ 
esis. 


When  questioned  about  the  strategies  used  to  develop  plans  in  the 
unaided  condition,  the  responses  centered  on  a  "carry  over"  technique. 

The  participants  indicated  that  they  learned  some  fundamental  principles 
from  the  aid  and  by  the  end  of  the  testing  were  developing  EMCON  plans 
manually  by  directly  applying  those  principles.  This  finding  again  sug¬ 
gests  the  utility  of  the  EWAR  aid  as  a  training  device. 

Generally,  the  participants  indicated  that  manually  developed  plans 
were  not  quite  equal  to  those  developed  with  the  aid.  According  to  the  par¬ 
ticipants,  the  fundamental  operation,  maximizing  information  denial,  could 
be  easily  performed  without  the  aid.  This,  of  course,  was  only  one  part  of 
plan  development.  The  remainder,  ensuring  adequate  surveillance  coverage, 
demanded  more  sophisticated  thinking  and  it  was  in  this  area  that  the  "mar¬ 
ginal  utility  of  the  aid  as  an  operational  tool”  was  apparent. 

When  the  participants  in  the  experimental  group  were  asked  if  they 
used  the  manual  techniques  and  methodologies  presented  during  the  training, 
they  generally  responded  in  the  negative.  This  held  in  spite  of  the  fact  that 
100  percent  said  that  the  manual  methodologies  were  adequate  and  sufficient 
to  the  task.  The  techniques  which  were  presented  were  held  to  be  too  cum¬ 
bersome  to  use  and  it  was  said  that  "once  you  had  some  experience  with  the 
aid  (usually  as  the  result  of  working  on  the  practice  problems  before  the 
'.tart  of  testing),  they  were  not  needed.  " 
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Assumptions  of  EWAR  Aid 

The  fourth  and  final  general  area  of  inquiry  was  related  to  several 
of  the  assumptions  on  which  parts  of  the  EWAR  aid  were  based.  The  first 
and  less  important  of  the  assumptions  was  concerned  with  how  threats  were 
tracked  and  defenses  targeted.  There  was  a  tendency  to  indicate  that  the 
targeting  and  tracking  assumptions  were  somewhat  unreasonable.  The  par¬ 
ticipants  seemed  to  think  that  the  aid  underestimates  the  number  of  threats 
that  will  be  destroyed. 

The  second,  and  somewhat  more  important  assumption,  was  con¬ 
cerned  with  the  targeting  of  threats  in  proportion  to  the  perceived  value  of 
targets.  While  the  responses  indicated  that  the  participants  agreed  to  a 
considerable  extent  that  the  enemy  would  probably  target  by  perceived  value, 
there  was  a  50-50  split  as  to  whether  or  not  the  enemy  would  derive  per¬ 
ceived  value  in  a  manner  analogous  to  that  employed  by  the  aid.  Half  of  the 
participants  thought  that  an  enemy  will  have  other  sources  of  targeting  in¬ 
formation  beyond  those  suggested  by  the  aid.  This  suggests  the  need  for 
revision  of  some  of  the  aid's  algorithms  and  for  realistic  algorithms  in 
any  decision  aid. 

When  probed  further  on  the  reasons  for  the  dramatic  shift  in  the 
comfort  level,  the  responses  seemed  to  be  related  to  "confidence.  " 

They  were  then  asked;  "what  if  the  second  plan,  which  resulted  in  sub¬ 
stantial  reductions  in  surveillance  coverage,  was  carefully  analyzed  by  the 
EWAR  aid  and  the  analysis  indicated  that  the  plan  reduced  expected  damage 
below  the  level  to  be  anticipated  with  all  the  emitters  on?"  Given  this,  the 
comfort  level  rose  to  the  "moderate"  level. 

Summary  of  Opinions 

It  appears  that  the  participants  thought  that  the  EWAR  aid  could  be 
of  value  to  fleet  operations.  There  was,  indeed,  a  good  deal  of  discussion 
about  the  appropriate  use,  if  any,  of  the  EWAR  decision  aid.  The  major 
negative  evaluation  of  the  aid,  and  that  which  suggests  its  inappropriateness, 
was  concerned  with  the  kind  of  targeting  information  which  may  actually  be 
available  to  an  enemy.  Many  participants  indicated  that  watching  the  ac¬ 
tivities  of  the  task  force  or  listening  to  their  communications  would  be  enough 
to  allow  inferences  about  the  location(s)  of  the  high  valued  ship(s). 


Most  positive  assessments  visualized  the  aid  as  having  a  multitude  of 
uses.  The  first  purpose,  and  the  one  most  consistently  stated,  was  to  use  the 
aid  during  the  training  of  all  EW  officers  and  task  force  commanders.  The 
second  potential  use  was  as  a  tactical  planning  device  which  would  be  used  to 
develop  contingency  plans  to  be  used  in  fleet  operations.  The  third  stated  po¬ 
tential  was  as  a  decision  aid  under  normal  fleet  operations.  To  meet  the  final 
purpose,  a  number  of  improvements  were  considered  to  be  necessary.  The 
most  important  improvement  is  the  implementation  of  a  system  that  automat¬ 
ically  suggests  the  one  or  two  best  plans.  This  would,  of  course,  be  especially 
useful  in  rapidly  changing  situations. 


One  variant  on  this  theme  was  the  suggestion  that  "devious"  plans 
which  hide  important  ships  may  not  be  appropriate  in  ail  situations.  Sev¬ 
eral  participants  thought  that  plans  which  depend  on  high  levels  of  informa¬ 
tion  denial  might  be  usefully  employed  to  get  an  intact  task  force  into  the 
area  where  a  mission  is  to  be  conducted.  Once  the  task  force  is  safely  in¬ 
to  the  area  of  operation,  the  need  to  hide  was  said  to  be  over  and  all  the 
emitters  might  be  turned  on.  This  suggestion  seems  to  be  a  reflection  of 
the  participants'  disbelief  that  the  enemy's  sources  of  information  would 
be  limited. 

Implications  for  Aid  Develonment 

The  present  work  has  suggested  a  number  of  implications  for  op¬ 
erational  decision  aid  development  and  design. 

From  the  point  of  view  of  aid  development,  the  most  primary  im¬ 
plication  seems  to  rest  on  the  suggestion  that  aid  developers  conceive  a 
causal  model  within  the  specific  decision(s)  they  are  aiding.  Such  a  model 
should  be  made  explicit  before  actual  aid  design  takes  place.  A  causal 
model  development  phase  within  the  aid  development  cycle  would  be  analogous 
to  the  conceptual  phase  of  the  equipment  development  cycle.  It  is  believed 
that  resting  aid  design  on  such  a  model  would  avoid  the  proliferation  of  aid 
features  and  displays  which  add  little,  if  anything,  to  the  utility  of  the  aid. 

At  the  least,  the  aid  developer  must  possess  some  implicit  model  before  he 
proceeds  with  his  detailed  design  and  development.  The  current  suggestion 
would  make  the  implicit  model  explicit.  We  do  not  advocate  modeling  the 
actual  mental  processes.  Rather,  we  advocate  providing  the  required  re¬ 
sults  of  the  mental  processes  in  the  form  which  is  logical  to  the  user.  Our 
contention  is  that  the  result  of  this  approach  to  aid  conceptualization  will  be 
aids  which  are  more  cost/ effective  and  which  are  more  valuable  from  the 
user's  point  of  view. 

One  area  in  which  the  EWAR  aid  was  said  to  require  improvement  was 
in  making  the  effects  of  variability  in  physical  factors  apparent  and  in  quan¬ 
tifying  surveillance  and  information  denial.  The  appropriate  means  for  rec¬ 
tifying  the  shortfall  is  not  at  all  obvious.  Certainly  any  decision  aid  could 
take  a  number  of  approaches,  either  at  molecular  or  molar  levels,  each  of 
which  derives  from  a  different  application  of  the  feedback  concept.  Accord¬ 
ingly,  in  the  context  of  the  EWAR  aid  and  at  a  molecular  level,  one  could 
extract  a  sensitivity  function  that  allows  examination  of  a  plan  in  regard  to 
variations  in  the  number,  direction,  or  type  of  threats,  or  the  destruction 
of  threats  in  regard  to  variability  in  the  number  and  type  of  radars  activ¬ 
ated,  etc.  On  the  other  hand,  from  the  point  of  view  of  information  denial, 
a  feedback  function  could  be  derived  to  analyze  the  effects  of  progressive 
changes  in  information  denial  on  the  fractional  value  of  the  task  force  re¬ 
maining,  or  some  other  appropriate  measure.  At  a  more  molar  level  and 
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again  in  the  context  of  the  EWAR  aid,  the  feedback  could  be  textual  in  nature 
and  a  result  of  a  critical  analysis  of  the  trade  off  aspects  of  an  EMCON  plan-- 
pointing  out  its  weaknesses,  how  it  might  be  improved  and  to  what  end.  Re¬ 
gardless  of  how  feedback  is  achieved  in  any  operational  aid,  it  should  not  only 
allow  for  insight  into  the  effects  of  certain  decisions  or  contingencies,  but  it 
should  also  make  clear  the  reasons  for  the  effects  of  certain  decisions  or  con¬ 
tingencies. 

Another  implication  is  the  requirement  for  aids  which  are  convenient 
to  use.  The  command  structure/ language  as  well  as  some  of  the  displays  in 
the  EWAR  aid  were  negatively  appraised  by  some  of  the  participants  in  the 
current  work.  User  convenience  features  such  as  menuing,  error  statements, 
prompting,  and  guided  use  are  well  within  the  current  state-of-the-art  and  should 
constitute  an  essential  ingredient  during  aid  design.  The  interface  between  the 
user  and  the  aid,  while  possibly  of  little  interest  to  aid  developers,  represents 
a  major  consideration  to  the  user.  Time  lost  and  the  frustration  involved  when 
the  user  attempts  to  make  the  aid  do  what  he  wants  it  to  do  represent  particu¬ 
larly  annoying  user  problems  and  negatively  impact  aid  acceptance  and  use. 

The  needs  for  basing  the  aid's  algorithms  on  realistic  considerations 
and  for  complete  algorithmic  consideration  of  all  important  impacting  vari¬ 
ables  was  made  apparent  on  the  basis  of  the  criticisms  of  the  EWAR  aid  that; 

(1)  an  enemy  would  base  targeting  information  on  information  other  than,  or 
in  addition  to,  perceived  value  as  derived  by  the  aid,  and  (2)  the  fraction  of 
task  force  remaining  algorithm  in  the  EWAR  aid  may  not  be  fully  comprehen¬ 
sive.  Such  pereeived  gaps  tend  to  cause  the  user  to  lose  confidence  in  a  total 
aid  when  only  one  or  two,  possibly  minor,  algorithmic  inadequacies  are  in 
evidence. 

The  greater  acceptance  of  "second  order"  (processed)  information 
as  compared  with  more  primary  (unprocessed)  information  speaks  to  a  pri¬ 
mary  feature  of  computer  driven  aids.  The  user  wants  the  aid  to  assist  him 
in  the  more  sophisticated,  complex  aspects  of  the  solution  process.  He 
thinks,  if  necessary,  he  can  easily  perform  the  less  complex  operations  him¬ 
self.  Accordingly,  any  aid  must  first  unburden  the  user  from  complex  infor¬ 
mation  processing.  Then,  and  only  then,  should  the  aid  become  involved  with 
the  more  routine  information  processing. 

Another  implication  is  related  to  the  suggestion  that  the  EWAR  aid  pro¬ 
vide  a  best  EMCON  plan  (or  two  plans--one  for  surveillance  coverage  and  one 
for  information  denial. )  The  participants  indicated  that  such  a  feature  would 
at  least  provide  a  point  of  departure.  Evidently,  the  users  wanted  the  aid  to 
save  them  as  much  time  as  possible.  We  note  here  that  the  participants  de¬ 
voted  less  time  to  achieving  problem  solutions  in  the  unaided  condition  as  com¬ 
pared  with  the  aided  mode.  However,  time  saving  was  not  the  only  considera¬ 
tion  involved.  Here,  again,  the  use  of  the  power  of  the  high  speed  computer 
system  seemed,  to  the  participants,  to  not  have  been  fully  taken  advantage  of. 


This  expressed  need  is  not  in  conflict  with  the  prior  study  (Siegel  &  Madden, 
1980)  which  reported  a  number  of  reservations  about  an  "expected  utility" 
function  which  was  designed  to  give  a  "best"  solution  to  a  strike  timing  prob¬ 
lem.  We  believe  that  the  participants  in  the  prior  study  considered  the  basic 
expected  utility  calculation,  which  was  simply  the  weighted  sum  of  lost  and 
destroyed  units,  to  represent  basic  information  whose  derivation  through 
computer  processing  was  nice,  but  not  necessary.  On  the  other  hand,  the 
EMCON  plans  suggested  by  the  participants  in  the  present  study  were  non¬ 
trivial  in  derivation  time  and  calculational  complexity. 

One  feature  of  the  EWAK  aid  which  seems  to  have  proven  its  merit 
was  the  hard  copy  capability  of  the  system.  This  feature  allowed  participants 
to  collect  quantities  of  information  from  a  variety  of  displays  and  a  number 
of  EMCON  plans  for  comparison.  This  situation  can  be  compared  with  the 
observations  from  our  prior  aid  evaluation  (Siegel  &  Madden,  1980)  in 
which  the  participants  spent  much  time  making  hand  copies  of  relevant  in¬ 
formation  and  reported  some  inability  to  make  comparisons  across  displays. 

It  would  appear  that  hard  copy  capability  may  be  a  necessary  component  of 
any  aiding  system. 

Some  of  the  difficulty  associated  with  the  introduction  of  any  operational 
decision  aid  rests  on  user  apprehension  about  aid  use,  aid  accuracy,  and  aid 
thoroughness.  It  seems  that  a  focused  user's  manual  should  be  made  available 
with  any  operational  decisions  aid.  If  users  have  such  a  manual  available  be¬ 
fore  they  use  an  aid,  procedural  problems  would  be  lessened  and  questions 
about  the  aid's  logic  would  be  answered  before  they  become  a  matter  of  user 
concern. 
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IV.  DISCUSSION  AND  CONCLUSIONS 


The  major  purpose  of  the  present  work  was  to  provide  an  eval¬ 
uation  of  the  EWAR  operational  decision  aid.  The  aid  was  designed  to 
provide  assistance  in  the  derivation  of  effective  radar  emission  plans 
by  task  forces.  As  such,  the  aid  possesses  a  wide  variety  of  features 
which  integrate  information  of  various  types  and  present  the  results 
of  the  integration  in  forms  which  can  be  used  to  provide  at  least  a  partial 
basis  for  EMCON  planning. 

Discussion 


The  evaluation  produced  evidence  from  a  number  of  sources  that 
suggests  that  the  EWAR  decision  aid  possesses  utility.  This  was  apparent 
from  the  variance  analyses,  the  perceived  utility  analyses,  and  from  the 
participants'  judgments  and  opinions.  In  addition,  the  multiple  regression 
analyses  and  the  usefulness  ratings  suggested  that  the  major  sources  of 
utility  were  based  on  the  trade  off  analyses  the  aid  allowed  and  on  the  infor¬ 
mation  cascading  provided  by  the  aid. 

The  analyses  of  variance  consistently  indicated  the  aid  level  factor 
tc  have  statistically  significant  effects.  This  held  for  the  fraction  of  own 
task  force  remaining  after  a  strike  (FRAC),  the  difference  between  the 
fraction  remaining  and  the  optimum  (DFRAC),  the  total  percentage  of  ra¬ 
dars  correctly  activated  and  deactivated  (PERCOR),  the  percentage  of 
radars  correctly  activated  (PERON),  and  the  percentage  correctly  deac¬ 
tivated  (PEROFF),  The  FRAC  and  DFRAC  analyses  were  particularly 
important  because  they  reinforce  the  finding  that  the  aid  helped  the  par¬ 
ticipants  to  select  plans  that  were  superior  on  a  measure  very  much  de¬ 
pendent  on  making  difficult  trade  offs.  The  other  scores  were  used  to 
examine  the  correspondence  of  the  participants'  and  "optimum"  criterion 
plans.  Using  the  aid  significantly  increased  the  probability  that  the  emitters' 
status  would  be  correctly  set.  Clearly,  the  use. of  the  EWAR  significantly 
increased  the  quality  of  the  EMCON  plans  whether  they  were  measured  by 
the  FRAC  scores  or  in  relation  to  some  criteria,  such  as  the  CFRAC  or 
PERCOR,  etc. 

The  multiattribute  utility  analysis  and  the  opinions  and  judgments 
of  the  participants  seemed  to  reinforce  the  notion  that  the  aid  was  gen¬ 
erally  a  useful  tool.  Across  seven  goals  of  the  EWAR  aid,  the  multi¬ 
attribute  utility  analysis  suggested  about  96  percent  goal  attainment.  In 
fact,  some  of  the  goals  were  possibly  overattained.  This  overattainment 
may  be  a  reflection  of  an  inappropriate  allocation  of  resources  or  it  may 
reflect  a  failure  on  the  part  of  the  developers  to  understand  the  importance 
of  comparing  alternative  plans.  With  a  large  set  of  plans,  such  compari¬ 
sons  could  become  quite  tedious.  It  may  be  that  the  pertinent  goals  of 
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the  aid  were  rated  highly  because  the  participants  implicitly  understood 
how  oppressive  manual  comparison  could  be. 


The  data  from  the  regression  analyses  and  from  the  usefulness  rat¬ 
ings  were  complementary  and  also  indicated  that  the  most  important  function 
of  the  EWAR  aid  was  to  provide  a  method  for  making  trade  offs  between  sur¬ 
veillance  coverage  and  information  denial.  There  were,  of  course,  some  prob¬ 
lems  inherent  in  the  aid  which  were  uncovered  by  the  evaluation.  Several  dif¬ 
ferent  sets  of  data  suggested  that  the  aid  did  not  adequately  sensitize  the  par¬ 
ticipants  to  a  variety  of  important  elements.  This  was  indicated  by  the  dif¬ 
ferences  between  the  PERON  and  PEROFF  scores,  and  by  the  underachieve¬ 
ment  by  the  aid  of  two  of  its  goals.  The  underattained  goals  were;  (1)  to 
help  quantify  the  surveillance  and  information  consequences  of  a  plan,  and 
(2)  to  understand  how  different  physical  factors  contribute  to  EMCON  planning. 

There  was  also  some  participant  disbelief  of  the  information  denial 
analysis  logic.  Most  participants  were  unsure  as  to  what  kind  of  information 
an  enemy  might  have  and  how  they  might  use  it  for  targeting  purposes.  It 
seems  that  this  problem  would  need  to  be  resolved  before  the  EWAR  aid  could 
be  given  attention  for  possible  fleet  use. 

Original  Hypotheses 

The  evaluation  was  designed  to  test  four  hypotheses.  The  first  hypoth¬ 
esis,  EMCON  plan  training  will  enhance  plan  quality,  is  not  accepted.  The 
training  failed  to  show  any  effects  in  the  most  important  variance  analyses. 
Although  there  was  evidence  that  training  may  have  had  some  effects  on  method 
of  approach  to  EMCON  planning,  these  effects  were  not  influential  on  plan 
quality  as  measured.  The  training  served  a  purpose  other  than  tutoring  the 
participants  in  EMCON  planning.  It  was  also  intended  to  teach  a  set  of  specific 
information  denial  rules.  It  was  expected  that  the  use  of  the  information  denial 
rules  would  enhance  plan  quality.  This  was  not  observed.  Knowing  the  rules 
did  not  give  the  experimental  subjects  any  detectable  advantage  in  EMCON 
planning.  In  fact,  the  control  subjects  deduced  the  rules  from  the  use  of  the  aid. 
Therefore,  neither  of  the  two  primary  elements  embedded  in  the  training  ex¬ 
erted  important  effects  on  plan  quality,  as  measured. 

The  second  hypothesis,  superior  EMCON  plans  will  be  produced 
when  the  aid  is  employed,  may  be  accepted.  The  results  of  each  of  the  variance 
analyses  lend  credence  to  the  acceptance. 

The  third  hypothesis,  the  problem  difficulty  will  not  affect  the  quality 
of  the  EMCON  plans  developed  with  the  aid,  is  also  supported.  However,  this 
support  must  be  qualified  because  difficulty  manipulation  similarly  did  not  ex¬ 
ert  any  effects  on  manually  developed  plans.  Perhaps,  difficulty  did  not  inter¬ 
act  with  aid  levels  because  manual  problem  solution  turned  out  to  be  an  applica¬ 
tion  of  knowledge  learned  when  using  the  aid.  The  transfer  from  the  aid  was 
mostly  in  terms  of  information  denial. 
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A  major  component  of  the  a  priori  difficulty  score- -the  compli¬ 
cation  component --was  a  measure  of  the  amount  of  information  processing 
which  would  be  required  to  produce  a  basic  information  denial  plan  for  a 
problem.  This  aspect  of  the  planning  was  highly  transferable  from  the 
aided  to  the  unaided  conditions.  Therefore,  the  difficulty  effect  was  miti¬ 
gated  across  conditions.  This  reasoning  also  may  provide  a  rationale  for 
the  failure  to  confirm  the  final  hypothesis,  a  high  correlation  will  be  noted 
between  the  a  priori  problem  difficulty  and  perceived  problem  difficulty. 

Conclusions 

The  evaluation  results  presented,  described,  and  discussed  through¬ 
out  this  report  tend  to  support  the  following  conclusions: 

1.  The  EWAR  decision  aid,  developed  by  Decision  Science 
Applications,  possesses  considerable  merit  and  po¬ 
tential  for  fleet  use  both  as  a  training  aid  and  as  an 
operational  support  instrument. 

2.  Before  any  additional  implementation  is  considered, 
modification  and  elaboration  of  certain  of  the  aid's 
algorithms  seems  indicated. 

3.  Similarly,  some  modification  of  the  EWAR  aid's  dis¬ 
plays  is  indicated  along  with  improvement  of  other 
user  oriented  features. 

4.  Application  of  the  EWAR  aid  could  result  in  about  a 
25  percent  increase  in  the  fraction  of  a  task  force 
remaining  after  a  strike  against  the  task  force. 

This  result  seems  to  be  fundamentally  a  function 
of  the  facilitation  of  trade  off  considerations  pro¬ 
vided  by  the  aid. 
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EWAR  Decision  Aid  Debriefing  Form 


Subject  Name _  Date _ 

Subject  I.  D.  _ _ 

Treatment  Condition _ 

The  first  set  of  questions  concerns  the  use  of  the  EWAR  aid  in  EMCON  plan 
development. 


1.  Would  you  say  that  the  EWAR  aid;  (a)  was  more  useful  in  developing  and 
evaluating  EMCON  plans  for  some  problems  than  for  others,  or  (b)  was  it 
equally  useful  for  all  problems? 

a  or  b: _  Why  so?  _ 


2.  With  this  usefulness  rating  scale  (show  card  with  scale)  rate  how  useful 


the 

following  information  was  in  the  development  of  the  EMCON  plans. 

Rating 

a) 

EMCON  plan  display 

1 

2 

3 

4 

5 

b) 

Surveillance  effectiveness  analysis 
(general) 

1 

2 

3 

4 

5 

Maximum  range  circles 

1 

2 

3 

4 

5 

Cumulative  detection  probabilities 

1 

2 

3 

4 

5 

Penetration  probabilities 
(surveillance  score) 

1 

2 

3 

4 

5 

c) 

Information  denial  analysis  (general) 

1 

2 

3 

4 

5 

Ship-to-blip  assignment  (INFO,  WHO) 

1 

2 

3 

4 

5 

Blip-to-ship  assignment  (INFO,  WHERE) 

1 

2 

3 

4 

5 

Probability  that  each  blip  is  being  emitted 
by  each  ship  (INFO,  DETAILS) 

1 

2 

3 

4 

5 

Probability  of  correctly  identifying  each 
ship  (INFO,  SUMMARY) 

1 

2 

3 

4 

5 

Comparison  of  perceived  to  true  value 
(INFO.  TARGET) 

1 

2 

3 

4 

5 

Information  denial  score  (INFO,  SCORE) 

1 

2 

3 

4 

5 

d) 

Trade-off  analyses  (General) 

1 

2 

3 

4 

5 

The  cumulative  probability-of-  detection 
and  perceived  value  lines 
(TRADE,  MAP,  THR  -  ) 

1 

2 

3 

4 

5 
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d)  continued 


Plot  of  surveillance  and  information 

denial  scores  {TRADE,  SCORES)  1  2  3  4  5 

Strike  outcome  scores  (TRADE,  STRIKE)  1  2  3  4  5 

I  would  like  you  to  rank  order  the  following  set  or  sets  of  information 
according  to  their  usefulness — from  most  to  least  useful. 

a)  First  rank  order  the  general  categories  of  information  that  were 
available. 

Rank 

EMCON  plan  displays  _ 

Surveillance  effectiveness  analysis 

Information  denial  analysis  _____ 

Trade-off  analysis  _ 

b)  Rank  order  the  surveillance  effectiveness  information. 

Maximum  range  circles  _ 

Cumulation  detection  probabilies  _ 

Penetration  probabilities  (SURV,  SCORE) 

c)  Rank  order  the  information  denial  displays. 

Probability  that  each  blip  is  being  emitted 

by  each  radar  (INFO,  DETAILS)  _ 

Comparison  of  perceived  to  true  value 

(INFO,  TARGET)  _ 

Information  denial  score  (INFO,  SCORE)  _ 


Rank 


d)  Rank  order  the  trade-off  information. 

Cumulative  probability -of-detection  and 
perceived  value  lines  (TRADE,  MAP,  THR  -  ) _ 

Plot  of  surveillance  and  information  denial 
scores  (TRADE,  SCORES)  _ 

Strike  outcome  scores  (TRADE,  STRIKE)  _ 

e)  Of  all  the  displays  listed  in  b,  c,  and  d  above,  what  were  the  five 
most  useful  displays? 

1 _ 

2 _ 

3  _ 

4  _ 

5  _ 

4.  Was  there  anything  confusing  about  the  tables  or  graphs?  _ 


Which  tables?  (Show  cards  with  an  example  each  table.) 
How  so? 


Which  Graphs?  (Show  cards  with  an  example  of  each  graph.) 
How  so  ? 


5.  What  additional  information  do  you  think  should  be  provided  by  the  aid? 
Is  there  any  additional  information  that  you  think  would  be  important  to 
provide  in  such  an  aid? _ 


Tables: 


p 


11-14.  The  next  few  questions  concern  how  you  developed  and  evaluated  EMCON 
plans  without  the  aid. 

11.  When  you  could  not  use  the  aid,  did  you  use  the  methods  supplied  during 
the  training  for  estimating  surveillance  effectiveness  and  information 
denial? 

Yes  or  No:  _ 

If  not,  what  methods  did  you  use? 


12.  Did  you  feel  that  these  methods  were  sufficient  to  the  task? 
Yes  or  No:  _______ 

How  so? 


13.  When  you  were  not  using  the  aid,  what  strategies  or  heuristics  did  you 
use  in  the  EMCON  plan  development  and  evaluation?  _ 


14.  What  were  the  differences  between  your  techniques  for  developing  EMCON 
plans  as  opposed  to  evaluating  plans  when  not  using  the  aid?  _ 


How  so? 


Was  there  some  information  that  you  used  for  developing  EMCON  plans 
that  you  did  not  use  in  evaluating  plans? 

Yes  or  No:  _ 

What? 


Why? 


15-20.  Now  I  would  like  to  question  you  about  what  you  perceived  to  be  the  important 
differences  between  developing  and  evaluating  EMCON  plans  with  and  without 
the  aid. 

15.  In  a  general  way,  do  you  feel  that  the  EMCON  plans  you  developed  with  the 
aid  were  any  different  from  the  plans  you  developed  without  the  use  of  the 
aid? 

Yes  or  No:  _ 
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15.  continued 

How  so?  _ 

I  ,  _ 

Was  one  technique  better  than  the  other? 

Yes  or  No;  _ 

How  so?  _ 

►  I  _ 

I  Qualitatively  or  quantitatively  different? 


‘  16.  What  was  the  largest  difference  in  developing  an  ENCON  plan  with  or 

‘  •  without  the  aid? 


I  17.  How  much  trust  did  you  have  in  the  information  you  calculated  yourself 

►  §  as  opposed  to  the  information  supplied  by  the  aid? 

Trusted  own  most;  _ 

Trusted  both  equally:  _ 

Trusted  aid's  most:  ________ 

Why?  _ 


18.  Do  you  think  the  use  of  the  aid  helped  you  to  have  any  more  confidence  in 
the  plans  you  developed  then  you  had  when  you  did  not  use  the  aid? 

Yes  or  No:  ________ 

How  so?  _ 


S 


19.  Would  you  feel  "comfortable"  in  using  EWAR  or  some  variant  of  it  under 
live  and  real  combat  conditions? 

Yes  or  No:  _ 

S  Why?  _ 


C 
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20.  In  a  live  combat  situation  would  you  be  surprised  if  the  actual  damage 
done  to  the  task  force  was  far  worse  than  that  predicted  by  the  EWAR? 

Yes  or  No;  _ 

Why? 


21.  The  aid  assumes  that  if  a  threat  is  detected,  after  an  initial  period  of 

90-seconds  during  which  time  the  threat  is  being  tracked  and  the  defenses 
targeted,  that  during  each  successive  120-second  period  half  the  threats 
will  be  destroyed.  To  what  degree,  do  you  think  that  this  is  a  reasonable 
and  valid  assumption?  Low  High 

Reasonable  degree;  1  2  3  4  5 

How  so?  _ _ _ 


Valid  degree: 
How  so?  _ 


Low 
1  2 


High 

3  4  5 


What,  if  any,  assumptions  do  you  think  would  be  more  valid  or  reasonable? 
Valid;  _ _ _ 


Reasonable: 


22.  Other  assumptions  incorporated  into  the  aid  are  that  the  enemy  will  target, 
in  accordance  with  the  perceived  value  of  targets  and  that  he  will  distribute 
his  fire  power  among  the  targets  in  order  to  optimize  destruction  in  accord¬ 
ance  with  the  perceived  value  of  the  blips. 

a)  To  what  extent  do  you  think  that  the  enemy  targets  by  perceived 
value?  Low  High 

Extent:  1  2  3  4  5 

Why?  _ _ _ 


b)  To  what  extent  do  you  think  that  the  enemy  distributes  its  fire  power 
so  as  to  optimize  damage  in  accordance  with  the  perceived  value? 

Low  High 

Extent:  1  2  3  4  5 
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22.  b)  continued 


Why? 


If  not,  on  what  basis  do  you  think  the  enemy  distributes  its  fire 
power  among  targets? 


c)  Do  you  think  the  enemy  derives  the  perceived  value  of  targets  in  a 
manner  analogous  to  that  used  by  the  aid? 

Yes  or  No:  _ 

Why?  _ 


If  no,  on  what  basis  do  you  think  the  enemy  evaluates  the  value  of 
targets? 


d)  Suppose  you  were  in  a  real  combat  situation  and  had  an  EMCON  plan 
tailor  made  for  the  task  force  and  the  mission.  The  plan  was  devel¬ 
oped  so  that  the  value  of  various  ships  will  be  misperceived;  i.  e. ,  a 
high  value  ship  should  be  perceived  to  be  a  ship  of  a  lower  value. 

This  was  done  under  the  assumption  that  the  enemy  would  allocate 
its  fire  power  in  relation  to  perceived  value  resulting  in  more  fire 
power  being  allocated  to  an  intermediate  value  ship  then  to  a  higher 
value  ship.  You  are  aboard  the  intermediate  value  ship.  What  would 
be  your  reaction? _ 


How  comfortable  would  you  feel  using  the  plan  if  it  resulted  in  a  reduc¬ 
tion  in  surveillance  coverage  of  the  task  force  from  what  would  be  ob¬ 
tained  with  an  ALL-ON  EMCON  plan  in  which  the  true  value  of  the  ships 
was  approximated  by  the  perceived  value? 


Low 

1 


Why? 


How  comfortable  would  you  feel  using  the  plan  if  it  resulted  in  a  sub¬ 
stantial  reduction  in  surveillance  coverage  from  that  afforded  by  the 
ALL-ON  plan? 


Low 

1 


22.  d)  continued 


Again,  consider  a  plan  which  altered  perceived  value  but  resulted 
in  a  substantial  reduction  in  surveillance  coverage  from  that  afforded 
by  an  ALL-ON  EMCON  plan.  How  comfortable  would  you  feel  using 
this  plan  if  a  careful  analyses  indicated  that  it  reduced  the  expected 
damage  to  the  task  force  below  the  level  expected  with  the  ALL-ON 
plan? 

Low  High 

1  2  3  4  5 


Why? 


If  you  would  feel  comfortable,  would  you  use  the  plan  in  a  real  com¬ 
bat  situation? 

Yes  or  No:  ________ 

Why?  _ 


Under  what  conditions  would  you  use  the  plan? 


23.  The  information  score,  supplied  by  EWAR  is  intended  as  a  pure  measure 
of  information  denial.  In  a  live  combat  situation,  how  much  confidence 
would  you  have  that  the  information  score  accurately  represented  the 
actual  information  denied  to  the  enemy? 


Low 

1 


Why? 


To  what  degree  would  you  be  willing  to  use  that  information  denial  score 
as  the  basis  for  making  an  inportant  EWAR  decision? 

Low  High 

1  2  3  4  5 


Why? 
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24.  The  surveillance  score  is  based  on  interception  probabilities  averaged 
over  various  threats.  The  score  is  intended  to  be  a  measure  of  pure 
surveillance  effectiveness.  How  much  confidence  would  you  have  under 
live  combat  conditions  that  the  interception  probabilities  are  valid? 


Low 

1  2  3 


Why? 


25.  Do  you  have  any  other  comments? 


26.  (Show  list  of  goals. )  Read  these  goals.  I  would  like  you  to  take  100  points 
and  distribute  them  among  the  goals  in  proportion  to  how  closely  each 
goal  was  achieved  by  the  EWAR  decision  aid.  That  is  if  in  your  estima¬ 
tion  the  EWAR  aid  achieved,  say,  goal  3  to  a  greater  extent  than  the  other 
goals  than  distribute  proportionally  more  points  to  it.  For  example,  a 
20/40/15/10/5/5/5/  distribution  would  indicate  that  the  second  goal  was 
achieved  to  twice  the  extent  of  the  first;  that  the  first  was  achieved  some¬ 
what  more  than  the  third  and  twice  as  much  as  the  fourth;  that  the  fifth, 
sixth,  and  seventh  were  the  least  achieved. 

Goals  of  the  EWAR  decision  aid. 


1.  To  facilitate  the  formation  of  hypotheses  about  how  best  to 
improve  a  current  plan 

2.  To  produce  better  EMCON  plans 

3.  To  help  quantify  the  surveillance  and  information  consequences 
of  a  plan 

4.  To  make  it  practical  to  examine  a  larger  set  of  alternative  plans 

5.  To  understand  how  different  factors  (threats,  radars,  etc.) 
contribute  to  EMCON  planning 

6.  To  improve  planner  confidence  in  the  quality  of  his  EMCON 
plans 


7.  To  facilitate  comparison  of  alternative  plans 
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27.  (Show  displays  by  goals  Table. )  I  would  like  you  to  partition  100  points 
for  each  display  across  four  general  goals  in  relation  to  how  much  the 
display  contributed  to  each  goal.  If  a  particular  display  was  more  im¬ 
portant  in  terms  of  one  goal  than  partition  proportionally  more  of  the 
100  points  to  it.  If  the  display  effectively  contributes  equally  to  each 
goal  than  partition  the  100  points  equally. 


Quantifying 

Understanding  Information  or  Comparing 


i 


I 
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LIST  OF  THE  EWAR  COMMANDS  AND  DISPLAYS 
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EWAR  Decision  Aid  Commands  and  Displays 


Command 


Order-of-Battie 


DIS,  EMCON 


Surveillance 

SURV,  CIR,  THR  -  Threat 
Name 


SURV,  CUM, 

THR  ■  Threat  Name 

SURV,  SCORE 


Information  Denial 


INFO,  DETAILS 


INFO,  DETAILS,  SHIP  =  Ship 
Name 

INFO,  TARGET 


INFO,  SCORE 


Description  of  Display 


Task  force  search  radar  order-of-battle 
and  which  radars  are  on/ off  for  an  EMCON 
plan 


Plots  maximum  detection  range  circles  for 
each  emitting  search  radar  for  the  specified 
threat 

Plots  cumulative  probability -of -detection 
contours  for  the  specified  threat 

Displays  surveillance  scores  for  each  threat 
system  and  for  an  average  system 


Lists  probabilities  that  each  blip  is  being 
emitted  by  each  task  force  ship 

Lists  probabilities  that  each  blip  is  being 
emitted  by  the  specified  ship 

Compares  "actual"  ship  values  with 
"perceived"  blip  values 

Shows  the  information  denial  score  for  the 
EMCON  plan  being  evaluated  and  for  the 
ALL  RADARS  ON  and  ALL  RADARS  OFF 
reference  plans 
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Command 


Description  of  Display 


Trade  Off  Commands 


TRADE,  MAP,  THR  =  Threat  Plots  cumulative  probability -of  detection 
Name  contours  for  the  specified  threat  and  draws 

blip  "perceived"  value  lines  below  ship 
symbols 

TRADE,  SCORES  Plots  surveillance  and  information  denial 

scores  which  have  been  "saved"  (Command 
is  SAVE,  SCORES  for  a  given  EMCON  plan) 

TRADE,  STRIKE,  Strike  Displays  results  of  a  simulated  "trade  off" 

Name  strike  (enemy  must  infer  blip  identities 

based  upon  radar  emission  data) 
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Operational  Commands _ Effect 


ON,  RAD=  Radar  Name,  SHIP= 
Ship  Name 

ON,  RAD=ALL,  SHIP=Ship 
Name 

ON.  RAD=  Radar  Name,  SHIP= 

»  ALL 

ON,  RAD=ALL,  SHIP=ALL 

OFF,  RAD=Radar  Name, 

SHIP=  Ship  Name 

OFF,  RAD=ALL,  SHlP=Ship 
Name 

OFF,  RAD=  Radar  Name, 

,  SHIP=ALL 

OFF,  RAD=ALL,  SfflP=ALL 

ON  for  OFF),  RAD=  First 
Radar  Name/ Second  Radar 
9  Name/Third  Radar  Name, 

SHIP=Ship  Name 

ON  (or  Off),  RAD=  Radar 
Name,  SHIP=  First  Ship  Name/ 
Second  Ship  Name/ Third  Ship 

•  Name 

SAVE,  SCORES 

SAVE,  EMCON  Plan  Name 

*  GET,  EMCON  Plan  Name 


Turns  on  the  specified  radar  on  the  specified 
ship 

Turns  on  all  radars  on  the  specified  ship 
Turns  on  specified  radar  on  all  ships 

Turns  on  all  radars 

Turns  off  the  specified  radar  on  the  specified 
ship 

Turns  off  all  radars  on  the  specified  ship 
Turns  off  specified  radar  on  all  ships 

Turns  off  all  radars 

Turns  on  (or  off)  more  than  one  radar  aboard 
the  specified  ship 

Turns  on  (or  off)  the  specified  radar  aboard 
more  than  one  ship 

Saves  Information  and  Surveillance  scores 

Saves  specified  EMCON  plan 

Sets  all  task  force  radars  as  specified  by 
the  EMCON  plan  named 
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APPENDIX  C 

DISCUSSION  OF  STRUCTURAL  EQUATION  MODEL 
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The  structural  model  is  illustrated  as  a  path  diagram  in  Figure  C-l. 
Straight  lines  in  the  model  indicate  a  causal  relationship.  The  direction  of 
the  causality  is  indicated  by  the  direction  of  the  arrows  on  the  straight  lines. 
The  bent  lines  containing  bidirectional  arrows  indicate  relationships  between 
exogenous  and  disturbance  terms.  The  model  contains  two  exogenous  vari¬ 
ables  and  three  endogenous  variables. 

The  exogenous  variables  are  not  the  concern  of  the  model  and  are  con¬ 
ceptually  distant,  but  necessary,  components.  The  first  exogenous  variable, 
X4,  represents  the  need  to  protect  the  task  force  and  it  was  assumed  that  the 
effect  of  this  need  was  to  maximize  surveillance  coverage.  The  second  ex¬ 
ogenous  variable,  Xg,  represents  a  need  for  security  and  it  was  assumed 
that  this  need  resulted  in  a  maximization  of  information  denial.  Both  ex¬ 
ogenous  variables  are  conceived  as  latent  variables  which  can  not  be  meas¬ 
ured  directly  and  must  be  estimated.  The  premise  behind  these  variables 
is  that  they  represent  two  divergent  responses  to  a  state  of  primary  motiva¬ 
tion.  This  primary  motivational  state,  which  is  not  incorporated  within  the 
model,  is  assumed  to  be  a  function  of  a  number  of  factors,  such  as  the  mis¬ 
sion  of  the  task  force,  the  psychological  make-up  of  the  task  force  command¬ 
er,  relevant  intelligence,  and  so  forth. 

Each  exogenous  variable  has  a  direct  effect  on  one  endogenous  vari¬ 
able.  The  "protection  need,  "  X4,  has  a  direct  causal  effect  on  X2,  the 
measure  of  surveillance  coverage  (the  surveillance  score)  forcing  it  toward 
a  maximum  value.  The  "security  need,  "  X5,  has  a  direct  causal  effect  on 
X3,  the  measure  of  information  denial,  forcing  it  toward  its  maximum. 

Obviously,  maximum  values  of  the  X2  and  X3  variables  can  not  lead 
directly  to  the  balance  the  trade  off  requires.  Therefore,  some  other  causal 
linkage  must  be  assumed  which  will  modify  these  maximizing  tendencies. 
Certainly,  direct  linkages  could  be  assumed  between  X4  and  X3,  and  X5  and 
X2,  representing  a  spread  of  the  effect  of  the  needs.  However,  the  frequent 
use  of  the  SAVE,  SCORES  command  (in  what  might  be  called  a  change-and- 
test  strategy),  led  to  the  supposition  that  the  causal  linkage  was  between 
X2  (surveillance  score)  and  X3  (information  denial  score),  each  having  a 
negative  effect  on  the  other.  This  follows  from  the  use  of  the  SAVE,  SCORES 
command,  which  resulted  in  the  presentation  of  the  surveillance  and  informa¬ 
tion  denial  scores;  that  is,  the  command  led  to  the  presentation  of  X2  and  X3. 
Presumably,  after  seeing  the  surveillance  and  the  information  denial  scores, 
the  scores  were  assessed  in  terms  of  their  effects  on  the  trade  off  scores 
and  strikes.  Apparently,  the  plan  chosen  by  a  participant  was  the  one  among 
the  alternative  plans  being  considered,  which  had  the  highest  trade  score, 
i.  e. ,  the  best  balanced  plan. 

The  causal  linkage  between  X2  and  X3  represents  the  point  in  the  mod¬ 
el  where  reciprocating  activities  resulted  in  a  modification  of  each.  The 
paths  of  each,  the  effect  of  X2  and  Xg  and  that  of  X3  and  X2,  are  represented 
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structural  equations 

Xj  =  Xj  +■  ^2  Xg  +  u 
X,=  Pa4X4  +  Pz 3X3+  V 
*1  5  P|3  *3  +  P|2  Xj  +  w' 


Figure  C-l.  The  trade  off  structural  equation  model 


in  Figure  C-l  by  the  double  lines  between  them.  Each  arrow  represents  an 
effect  in  the  opposite  direction.  It  is  further  assumed  that  X2  and  X3  both 
affected  Xj_,  the  measure  of  the  achieved  balance,  the  trade  score. 


The  model  attempts  to  account  for  the  variability  in  each  of  the  en¬ 
dogenous  variables  Xl,  X2,  and  X3,  as  functions  of  causal  relationships. 
However,  total  variance  can  never  be  completely  accounted  for  by  a  model. 
The  relationships  are  therefore  assumed  to  be  attenuated  by  disturbance 
terms.  The  disturbances  shown  in  Figure  C-l  (u,  v,  and  w)  represent  other 
correlated  sources  of  variability  in  the  endogenous  variables  not  specified 
in  the  model  (Duncan,  1975). 

The  model  which  is  nonrecursive  and  fully  identified  may  be  repre¬ 
sented  by  three  structural  equations: 

x3  *  P35x5  +  P32x2  +u‘ 

X2  "P24x4  +  P23x3  +v' 

X-l  “  Pl3x3  +  P12X2  +w' 

where  the  p's  are  the  structural  or  path  coefficients  and  where  because  of 
the  correlation  in  the  distrubance  terms: 

u'  “  P3UU  +  p32p2vv 
v'  -  P2vv  +  P23P3uu 
w'”  Piww  +p13u'  +  pi2v' 


In  structural  equation  models,  a  simplifying  assumption  is  that  the  path 
coefficients  for  the  direct  effect  of  the  disturbances  equals  one  (Duncan, 
1975).  Therefore,  the  disturbances  terms  can  be  rewritten; 

u'  =  u  +  p9„v 


v'  =  pv  +  P3uu 

w'  =  w  +  p13u'  +  p12v'. 
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